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from: Steve Johnson 
MH 45415 
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A1 Aho and Ravi Sethi are writing a book on compiler 
construction and would like to try it out on some of us. 

They propose teaching a course on compiler construction, 
primarily to members of 45415 but with a few additional 
people as space permits. The course will be graduate level, 
and, knowing the authors, will concentrate on principles 
rather than implementation techniques. There will be 
exercises, but no formal grades; however, serious students 
willing to stick it out to the end are encouraged, the 
better to give feedback about the book. 

The first meeting will be Thurs, Dec 15, at 10AM in 2C-123. 
After a brief organizational meeting, the first day's topic 
will be discussion of a simple one pass compiler. Further 
meetings will be every other Wednesday at 9:30, starting on 
Jan 11, 1984. We expect that the place of meeting will move 
to Summit when we do. The dates are chosen so as not to 
conflict with the 454 Seminars. 

If you are interested, please mail to Martha Petruce 
(mh3bs5!martha), or call 5777. Enrollment will be limited 
if it grows beyond bounds of reasonable Xeroxing of course 
materials. 


Copy to 

Members of 45415 
Supervision, 454 
A. V. Aho 
J. Feder 
R. Sethi 
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COMPILERS: Principles, Techniques, and Tools 


Meetings on alternate Wednesdays, 9:30-11:30, starting January 11 


Proposed Schedule 


Date 

Room 

Speaker 

Subject 

Dec. 15 

2C-123 

R Sethi 

A simple one-pass compiler 

Jan 11 

6B-201 

A Aho 

Lexical analysis 

Jan 25 

6B-201 

A Aho 

Syntax analysis 

Feb 08 

2C-123 

T Reps, Cornell 

Generating language based environments 

Feb 21* 

3A-216 

S C Johnson 

Parser generators 

Mar 07 

2C-123 

R Sethi 

Attribute grammars 

Mar 21 

2C-123 

R Farrow, Columbia 

Attribute grammar based compilers 

Apr 04 

2C-123 

R Sethi 

Type checking 

Apr 18 

2C-123 

Quinn/Wetherell 

Ada front end 

May 02 


R Sethi 

Scope and extent 

May 16 

SF 

M D McHroy 

Storage allocation 

May 30 

A Aho 

Abstract machines 

Jun 13 

4-\\°\ 

A Aho 

Code generation 

Jun 27 

C Fischer, Wisconsin/Cornell 

Retargetable code generation 

Jul 11 

Jul 25 

A Aho 

Code optimization 

Experience in optimization 


Stave Johnson’s meeting will he from 3‘00-5'00pm.. on Tuesday. Feb. 21. 



COMPILER COURSE PROJECTS 


Here are some random projects geared to a UNIX audience dial can be done by small groups to 
test out die concepts of this course. Anyone interested is encouraged to talk to the instructors 
before plunging in. 

I Reimplement LEX. 

2. Write an incremental LR parser. 

3 Devise more effective error recovery routines for LR parsing. 

4 Construct a tool that combines lexical and syntactic analysis specifications. 

5 Write a program that computes the inverse GOTO function for YACC tables. Also write a 
program to associate a shortest viable prefix with each YACC state. (Useful in debugging 
grammars.) 

6 Construct an error-correcting parser far C 

7 Construct a programmable pretty printer for C 

8 Write a type-checker generator. 

9 Define and implement intermediate language that can be used for several current languages. 

10 Construct a peephole optimizer for intermediate language. 

II Construct a peephole optimizer for assembly language. 

12 Evaluate the effectiveness of various optimizations for C 

13 Evaluate the effectiveness of a table-driven code generator far C 

14 Construct a timing profiler for C 
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CHAPTER 2 


In trod action 
to C om pliers 


Compiling obeys the 80-20 rule: 80% of the job can be done with 20% of the 
effort. Clearly this statement is not prase, but the fact remains that simple 
te c hn iques go a long way towards solving translation problem that arise in 
practice. Some of the basic techniques used to compile programming 
languages are presented in this chapter. The rest of the book discusses exten¬ 
sions and more advanced techniques. 

Underlying the presentation of concepts in this chapter is the development 
of a compiler for a little language containing expressions and while-loops. 
Considering a complete compiler, even if it is for a little language, allows the 
interaction between its various parts to be illustrated. Altho ugh the little 
language is simple, many compilers, including the Pascal P compiler, look a 
lot like the one in this chapter. 

The emphasis in this chapter is on concepts that are widely applicable, with 
working programs showing the realization of the concepts. As working pro¬ 
grams must be in some implementation language, we use C, although the pro¬ 
grams can be transcribed straightforwardly into other languages. rwniit 
peculiar to C are ex plaine d as necessary. 

2.1 A One-Pass Compiler 

We shall construct a compiler that translates expressions into code for a stack 
machine. We begin by considering a simple special case: expressions consist¬ 
ing of lists of digits separated by minus signs; e. g., 2-3-5. As the basic 
ideas become dear, we extend the language to include more general expres¬ 
sions and flow-af-control constructs. During the construction of our compiler, 
we discuss: 

1. Syntax and Semantics. A language to be compiled is defined by describing 
what programs look like, the so-called syntax of the language, and what 

These naes are far the exclusive use of students in tte AT&T Bell Laboratories Special 
Came “Compilers: Principles, Techniques, and Tods,” December, 1983 to June, 1984. 

No part of these notes may be copied cr reproduced without written pemissicn of tl* au¬ 
thors: Alfred V. Aho, Ravi Sethi, and Jeffrey D. Tillman 
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programs mean, the semantics of the language. In this chapter we than 
use context-free grammars to specify the syntax of a language and use 
informal descriptions and examples to suggest the semantics. Our first 
program sickly recognizes if its input is an expression consisting of a list 
of digits separated by minus signs. The recognizer is then mrdifieH to 
translate expressions into postfix form in which operators appear after 
their operands. Our final translator for die language with ex pressions and 
flow-of-control statements is fanned by extending this r eco g nizer in a 
straightforward way. 

2. Syntax-Directed Translation. Besides specifying what progr am s look like, 
the syntax of a language can be used to impose structure on programs that 
can guide the translation of p re gra ms. This technique, known as syntax- 
directed translation, is very helpful in organizing a compiler and will be 
used throughout this chapter. The early part of this chapter deal s with a 
parsing technique far analyzing the syntax of a program. We discuss how 
translation can be done during syntax analysis. 

3. Lexical Analysis. The input to a compiler is a stream of characters consti¬ 
tuting the source program. This stream is converted by a lexical analyzer 
into a stream of logical units, called tokens, that farms the inp ut to the fol¬ 
lowing phase, the parser. Lexical analysis will be discussed in detail in 
Chapter 3. In this chapter we shall construct a rudimentary lexical 
analyzer that allows white space in the form of blanks, tabs, and newline 
characters to separate the tokens of a progr a m. The lyrical analyzer also 
collects consecutive characters into identifiers and keywords. 

4. Symbol Tables. In its simplest form, a symbol table is a data structure for 
keeping track of identifiers that have been seen. Additional information 
may be added to the symbol table as needed. In particular die symbol 
table in this chapter also keeps track of keywords like begin and while 
by treating them as reserved identifiers. 

5. Abstract Machines. As we indicated in Chapter 1, the front end of a com¬ 
piler is insulated from die details of a target marhine by doing code gen¬ 
eration in two stages. Initially, intermediate code for an abstract marhine 
is produced. This intermediate code is then tr anslat ed into code for a tar¬ 
get machine. The compiler in this chapter is basically only a front end. 
As outjxit, it produces code for a stack machine, which is an abstract 
machine that performs computations using a stack. A stack machine was 
chosen since code can be generated easily for it The stack machine code 
can also be used as input to a code generator that will produce object code 
for a particular target machine. Code generation will be dism^yd in detail 
in Chapter 9. 
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2.2 Syntax Definition 

Context-free gra mm a r s are a precise notation for specifying the syntax of a 
language. Net only do they specify which sequences of symbols constitute a 
program, the structure they impose on programs can be exploited to translate 
programs. The compiler in this chapter is organized around a grammar. 

Grammars, which are also known as ENF (Backus-Naur Form) notation, 
will be introduced by considering specifications for expressions consisting of 
lists of digits separated by minus signs. Lists of items appear frequently in 
programming languages. Obvious examples indude parameter lists and 
sequences of statements. In a sense, arithmetic expressions are just *vstrd 
lists of items separated by arithmetic operators. 

Lists of digits separated by mi mm signs, of which 

7 

1- 3 

2- 3-5 


are e xampl es, are composed of symbols that we shall call tekeru or terminals. 
The tokens in these expressions are the mimi* sign -, die digits 0, 1 , and so 
on, through the digit 9. Not all sequences of tokens form lists of digits, since 
two digits must be separated by a minm sign. 

The sequences of tokens which are permitted will be specified by rWimnn 
abstractions calle d nonterminals o r var iables, each of which represents certain 
tokens or sequences of tokens. Fa example, we might define the nonterminal 
digit to stand far any of the tokens 0, 1, • • • 9. 

To specify the definition of nonter minals we use productions, which are 
statements such as 

digit -* '0' 

In general, a production has a nonterminal cm the left, an arrow, which we 
may read as “can have the farm,” and a sequence of tokens and/cr terminal 
symbols, called the right side of the production. Tokens on the right side of a 
productions are enclosed in single quotes. Thus, digit can be completely 
specified by the ten productions 

digit •* 'o' 

digit ■* * 1 ' 

digit - '9' 

For notatianal convenience, productions with the same nonterminal on the left 
can have their right sides grouped, with the alternates separated by the symbol 
|, which we read as “or.” Thus the ten productions far digit can be written 
on one line as 

digit - '0' | '1' | • • • | '9' 

Now we can define the nonterminal list to represent lists of digits 
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separated by tnirwis signs. Since a single digit by itself is a list, one production 
for list is 

(1) list — digit 

It is also true that if we take any list and follow it by a minus sign and 
another digit we have a new list. We can exp re s s this fact with the production 

(2) list ■* list digit 

It turns out that productions (1) and (2) are all we need to define the lists we 
are interested in. For example, we can deduce that 2-3-5 is a list as fol¬ 
lows. 

a) 2 is a list by production (1), since 2 is a digit. 

b) 2-3 is a list by production (2), since 2 is a/iif and 3 is a £git. 

c) 2-3-5 is a list by production (2), since 2-3 is a list and 9 is a £git. 
Formally, a context-free grammar consists of four components: 

1. Aset of tokens. 

2. A set of nonterminal symbols. 

3. A set of productions. 

4. A designation of one of the nonterminals as die start symbol. 

A string of tokens is a sequence of zero or more tokens. The token strings 
defined by the start symbol form the language defined by the grammar. 

Conventionally, we shall specify grammars by listing the pro du ctio n s, with 
the productions far the start symbol listed first. We use italics for nontermi¬ 
nals and tokens will usually be singly quoted, although any nomtahazed name 
may be msmneri to be a token. 

FViwrU 2.1. The gramm ar for lists of digits that we developed above may 
be written 

list ■* list '-' digit | digit 

digit - '0' | '1' | • ' • | '9' D 

FT.mpi* 2.2. A somewhat different sort of list is a list of statements 
m»pwrnted by semicolons, found in Pascal begin-end blocks. One nuance of 
sudi lists is that an empty list of statements may be found between begin and 
end. We may start to develop a grammar for begin-end blocks by induding 
the productions: 

block - 'begin* optstmts 'end' 

optstmts — stmt-list \ t 

stmt-list — stmt-list stmt | stmt 

Note that the second possible right side for optstmts is e, which stands for 
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the empty string of symbols. That is, optstmts can be replaced by nothing at 
aD, so a block can consist of the token string begin end. We have not 
shown the productions for stmt . Shortly, we shall disniw the ap pr o p r iate pro- 
ductions for the various kinds of statements, such as if-statements, assignment 
statements, and so on, that we will be interested in. □ 


Fane Trees 

The structure imposed by a gr ammar on a string of tokens is neatly depicted 
by a tree- Figure 2.1 shows the tree structure imparted to the list 2 - 3-5 by 
die grammar of Example 2.1. 

Trees like the one in Fig. 2.1 are called parse trees. Formally, a tree 
whose nodes are labeled by the tokens and nonterminals of sonv*. context-free 
grammar is a parse tree if: 

1. The root is labeled by the start symbol. 

2. Each leaf is labeled by a tnln»n or by e. 

3. Each interior node is labeled by a nonter minal 

4. If A is the nonterminal labeling some interior node nnH x j, Xi, X 
are die labels of the children of that node from left to right, then 
*i *2 * • • *■ is the right side of some production for A. 

Example 23. In Fig. 2.1, the root is labeled list, which is the start symbol of 
the grammar in Example 2.1. The children of the root are labeled, from left 
to right, list, -, and digit. Note that 

list - list digit 
is a production of Example 2.1. □ 

On re a di n g the leaves of a parse tree from left to right we get the string 
generated or derived from the starting nonterminal by tte parse tree In Fig 
2.1 the generated string is 2-3-5. Another definition of the language gen^ 
crated by a grammar is as the set of strings that can be generated by sane 
parse tree. F i n d in g a parse tree for a given string of tnfrrm j s cafled parsing 
that string. 0 

Ambiguity 

We have to be careful in talking about the structure of a string according to a 
grammar. While it is dear that each parse tree derives exactly the string read 
off its leaves, a grammar can have more than one parse tree generating a 
given string of tokens. Such a gr amm ar is is said to be ambiguous. 

* j n m I | i < ‘ 2 - 4 * Suppose we did not distinguish between digits and lists as in 
Exa mpl e 2.1. We could have written the gr ammar 

string - string string \ '0' \ 'V | ••• | 
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list 



Fig. 2.1. Parse tree for 2-3-5 according to die grammar for list. 
Henceforth we will not attempt to place all the leaves at the same 
level. 

Merging the notion of digit and list into the nonterminal string makes superfi¬ 
cial sense, because a single digit is a special case of a list. 

However, Fig. 2.2 shows that lists like 2-3-5 have more than one parse 
tree according to the new gramm ar. The two trees for 2-3-5 correspond to 
the two ways of parenthesizing the Hst (2-31-5 and 2-(3-5). This 
second parenthesization allows us to interpret die expression 2-3-5 
incorrectly as 2-0-5), which has the value 4. Note that the grammar of 
Example 2.1 does not permit this incorrect interpretation. □ 


string string 

/\\ /\\ 

string - string string - string 

/l\ I I /l\ 

string - string 5 2 string - string 


Fig. 2.2. Two parse trees for 2-3-5. 

Sint*, a string with more than ow parse tree usually has more than one 
translation, we frequently restrict our attention to unambiguous grammars, or 
to ambiguous grammars with additional rules to resolve die ambiguities. 


Syntax of Progr amm ing Languages 

We conclude this section by illustrating in more detail how grammars can be 
used to specify the syntax of pro grammi ng language constructs. In the next 
two examples we present examples for Pascal-like statements and arithmetic 
expressions. 

Fjampi# 2.5. Syntax of statements. Keywords make it very easy to identify 
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statements in most languages. With the exception of iwi pmmt. and pro¬ 
cedure calls, all Pascal statements begin with a keyword. The syntax of seme 
Pascal statements is given by the following gr ammar 

stmt — identifier exp 

| 'if' exp 'then' stmt 
| 'if* exp 'then' stmt 'else' stmt 
| 'while' exp 'do' stmt 

| 'begin' optstmts 'end' 

The nonterminal optstmts generates a possibly empty list of statements 
separated by semicolons using the productions presented in Example 2.2: 
optstmts - stmUist \ t 

stmt-list — stmt-list ';' stmt | stmt O 

Example 2.6. Expressions are lists of lists. The transition from lists of digits 
t° expressions is very easy. The expressions 2*3 and 2*3*5 are lists of 
digits separated by • symbols. Similarly, 2*3*5 is a Hst of digits separated 
by + signs. The interesting part is combining the various operators. 

Consider expressions with both addition and multiplication. 2 + 3 * 5+7 
contains the following subexpressions: 2, 3*5, 7. Calling subexpres¬ 
sions terms , the structure of 2*3*5+7 is given by term ♦ term * term-, i.e., a 
hst of terms separated by + symbols is an erpn-wrm 

For completeness we give a basic grammar for expressions, deferring dis¬ 
cussion until Section 2.6 where expressions are gnwnirwd in %rmr detail. 

exp - exp '+' term | exp term \ term 

term - term '*' factor | term '/' factor | factor 

factor - constant | identifier | '(' exp ')' 

The details of constant and identifier will be left unspecified for the moment. 
Notice that any parenthesized expression is a factor, so with parentheses we 
can develop expressions that have arbitrarily deep nesting (and also arbitrarily 
deep trees). Thus, there is more structure in arithmetic gx pmstinm than in 
simple lists of dements. □ 

Example 2.7. Associativity of Operators. The minus operator is said to be 
left-assodative si-ice the only correct parenthesization cf 2-3-5 is (2-3)-5 
Sunilarly, the assignment operator, -, in C is said to be right-associative, 
since the only correct parenthesization cf a«b»c is a»(b-c). 

Strings like a*b-c can be generated by the grammar: 

right - letter right | letter 
letter - 'a' | 'b' | • • • | 

The contrast between die grammars for left-assodative operators so far in 
this section and this new grammar for a right-assodative operator is suggested 
by Fig. 2.3. The parse tree on the left is a redrawing of the tree in Fig. 2.1 to 
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wnph^iTy the fact that the tree grows down towards Ac left. Note how the 
parse tree on the right for a*b-c grows down towards the right. □ 


list 

/l\ 

list - digit 

'l\ I 

- digit 5 

I 

3 


right 

/\\ 

Utter » right 

I /l N 

a Utter - 

I 

b 


right 

I 


list 

I 

digit 

1 I 

2 c 

Fig. 2.3. Parse trees for left- and right-associative operators. 


23 Syntax-Directed Translation 

The front end of a compiler analyzes die syntactic structure of a source pro¬ 
gram. More precisely, it determines if a parse tree exists for the program 
and, if so, it deduces implicitly or explicitly what that structure is. The results 
of the analysis can be recorded either by explicitly constructing a parse tree or 
by translating the program into same intermediate farm as the analysis takes 
place. A popular intermediate form is code for an abstract stack ma c hin e, and 
this is the farm our compiler will use. 

A syntax-directed translation of a language can be specified by a tta c hing a 
semantic rule to e ac h production of a context-free gr amm a r for that language. 
A context-free gr ammar with such semantic rules is often called a syntax- 
directed translation scheme. For illustration, we shall present a syntax-directed 
translation for translating expressions into postfix notation. 

The ordinary infix way of writing the difference of two expressions x and 
y is with the operator in the middle: x-y. The postfix notation for the same 
expression {daces the operator after the operands: xy-. No parentheses are 
nffdfd in postfix notation because the position and arity (number of argu¬ 
ments) of the operators permits only one decoding of a postfix expression. 
For example, the postfix representations of (2-31-5 and 2-(3-5) are 
23-5- and 235—, respectively. Our main reason for considering postfix 
notation here is that it is very close to code for a m ac h i n e that performs all 
computations n«mg a stack. a machine forms a useful intermediate code 
for compilers and we shall discuss its use in Section 2.8. 

Syntax-Directed Translation Schemes 

In a syntax-directed translation scheme we associate one or more attributes 
with each nonterminal of a context-free grammar. For example, we might 
declare that a nonterminal A has an attribute t. The attribute can represent 
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any quantity, for example, a type, a string, cr a location. 

These attributes are used to define a translation in the following 
Suppose we have constructed a parse tree for some input string. At each node 
of the parse tree, we compute a value fee each of the attributes associated 
The value of attribute t at node A will be denoted t[A]. If 
““ “sed at node A , the value of each attribute at node* is 
computed using the semantic rule associated with this production. A raise 
tree with attributes values at each node is called an awwiate/parse tree.*^e 
value of some attntxite of the root of the parse tree is often taken to be the 
translation for the string generated by that tree. 

Emnple 2.8. Let us develop a syntax-directed translation scheme for 
transhm^ mpressions consisting of lists of digits separated by minus signs 
into p ostfix fo rm, using the grammar of Example 2.1. With each nonterminal 
of the grammar we shall associate a string-valued attribute r, whose value will 
in a 8tring t ° kcns 8 ma ' atcd ty that nontermina] 

A single dpt is already in postfix form, so the semantic rules fer single 
digits simply echo the productions. 

digit - *0' { ([digit] - '0' } 

digit ~ 'V i ([digit]- 'V ) 

digit - '9' { {[digit] - '9' j 

Each node in a parse tree corresponds to a production; the semantic rule asso¬ 
ciated with the production is applied to compute the value of t at the node 

Fr °“ * c <* ^ t attribute at a node fer digit is deter¬ 

mined by the child of the digit node. 

Let us now consider the translation of lists. When a list is a single digit 
its translation is simply that digit: 

tor - digit { r[/iir] - {[digit] y 

In a list with a minus sign, the minus sign is moved into the proper postfix 
posinon using the following semantic rule: (the subscripts on Zm are usedmly 
to distinguish the two lnstarv-r* of y 

tor 0 -* listi digit 

{ r[/£rr 0 ] - r[toj] {[digit] > 

This syntax-directed translation scheme is an inductive spedficaticn of the 
bsts form * A semantic rule may be 

^°,S Ch mtelQt a fra if we work bottom up from the 

leaves of the parse tree toward the root. For exampl e Fig. 2 4 shows the 
Rattan tofc p™ «» fa 

r^ is taken to be the postfix translation of tte string generated by the parse 
A nonterminal may have many attributes. For exampie, along with the 
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Hg. 2.4. An annotated parse tree showing attributes evaluated at each node. 


translation of a programm ing language statement into ma c hin e code, attributes 
specifying the size of the translation or the location of die first instruction in 
the translation are needed. 

In Example 2.8, the attribute t of the nonterminal on the left hand side of 
each production was defined in terms of the attributes of the grammar sym¬ 
bols on the right Hand side of the production. Such attributes are called syn¬ 
thesized. The term “synthesized” suggests that the translation of a nontermi¬ 
nal ina parse tree is built up from the attributes at the nodes below it. Other 
types of translation rules will be discussed in Chapter 5. 


Implementation of Syntax-Directed Translation Schemes 

A rule like 


listi digit 

{ t[list 0 ) - 


t[listi] t[digit\ > 


specifies the relationships between the translations of the nonterminals, and is 
of the order in which t[listi] and t[digit] are determined. One 
way to im ptffnvnt a syntax-directed translation scheme is to replace each 
semantic rule by a fragment of code that computes the values of the attributes 
in the semantic rule. For example, a straightforward implementation of the 
syntax-directed translation scheme in Example 2.8 wtxild actually construct 
and concatenate rf/ufi] and t[digit]. Because of Jae specialized form of the 
semantic rules in Example 2.8, the translation can be done without staring any 
strings, simply by printing symbols during the bottom-up construction of a 
parse tree. 

damply 2.9. Figure 2.5 shows the parse tree for 2-3-5 with printing 
actions attached to the nodes. The actions for <Bgit are the obvious ones: 


digit - '0' 

{ print '0* } 

digit - *V 

{ print } 

digit - *9' 

{ print *9' } 
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Since digits in list x and digit have already been printed, only the minm sign 
remains to be printed in the following rule: 

list 0 - listx digit { print } 

Finally, nothing is printed in the rule 
list - digit { } 

The braces {} can be dropped if there is no translation rule, o 

We shall call a code fragment that is executed whenever a production is 
used a semantic action . As we shall see, we may wish to place semantic 
actl0ns anywhere within the right side of a production, and we may even wish 
to embed several actions at different points within one right side. 

Ordering of Semantic Action 

Imphat in the decision of what to print is an ordering on ncmantir actions. 
For example, the proper order of the print actions for Fig. 2.5, and other 
annotated parse trees constructed from the rules of Example 2.9, is obtained 
by a postorder traversal of the tree. Thus, for the rule 

list 0 - listx digit { print ) 

correct translation requires that all the printing in the subtree for listx be draw 
before that for digit ; the minus sign must of course be printed last. That is 
exactly what happens if we perform print actions in postorder; the action asso¬ 
ciated with the node corresponding to list 0 follows the action associated with 
digit , which follows all actions associated with the node fer list : and its des¬ 
cendants. 

As a general rule, when specifying translations using semantic actions we 
shall assume a left to right order of evaluation. That is, an action is per¬ 
formed after all the terminals and nonterminals to its left in the right side of 
the production have been recognized on the input, and before any symbols to 
its right have been recognized. This assumption is easily satisfied in practice 
because strings are generally processed from left to right in a “greedy” 
fashion; i.e., as much of a parse tree is constructed as possible before the next 
input token is read. 

The above assumption generalizes to actions that ~xur in the middle of a 
production. In 

list 0 - listx { print } digit 

the actions within the subtree for listx *re performed first, then the minus sign 
is printed, and finally the action in the subtree for digit is performed. A sim¬ 
ple inductive proof on the height of the node in the parse tree corresponding 
to list 0 shows that this rule (together with the rules for digit) limply echoes its 
input. 
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list (print'-'} 



list (print'-'} digit (print'5'} 



list digit (print'3'} 

I I 

digit (print'2'} 3 

1 

2 

Fig. 2.5 Actions during the postfix translation of 2-3-5. 

2.4 Recanhre-Descent Parsing 

Parsing is the process of determining if a string of tokens can be ge nerated by 
a gram mar. In discussing parsing it is helpful to diink of a parse tree being 
constructed, even though one-pass compilers do not actually construct such a 
tree. So we shall view parsing as the process of constructing a parse tree for a 
program. 

For any context-free grammar there is a parser that takes at most 0(n J ) 
time to parse a string of length n. But cubic time is too much. linear algo¬ 
rithms suffice to parse essentially all grammars that arise in practice. Given a 
notation or programming language, we can generally construct a grammar for 
the notation that can be parsed quickly. Programming language parsers 
almost always make a single left to right scan over the input, looking ahead at 
one token at a time. The advantage of this approach is that the entire input 
stream of tokens does not have to be kept in main memory. 

Most parsing methods fall into one of two classes, called the top-down and 
bottom-up methods. These terms refer to the order in which nodes in die 
parse tree are constructed In the for mer , construction starts at die root and 
proceeds towards the leaves, while in the latter construction starts at the 
leaves and proceeds towards the root. The popularity of top-down parsers is 
due to the fact that efficient parsers can easily be constructed by hand On 
the other hand, bottom-up parsing can handle a larger of grammars and 
translation schemes so software tools for generating parsers directly from 
grammars have tended to use bottom-up methods. 

This section begins with a general discussion of recursive-descent parsing 
which has been used in many production compilers. A concrete example is 
then given by constructing a parser for lists of digits. In the process we also 
show some of the techniques used to adapt a grammar for top-down parsing. 
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Introdoctko to Recursive-Descent Parsing 

A subdass of top-down parsers called recursive-descent parsers is easy to 
i mp le m e n t in a language that supports recursive procedures. Before getting 
into the details of this parsing method, let us consider some ^ampi^ 
focus on statements defined by the following grammar: 

stmt — identifier ' exp 

| 'if' exp 'then' stmt 
| 'while' exp 'do' stmt 

| 'begin' optstmts 'end' 

optstmts - stmUist | t 

stmt-list — stmt-list ';' stmt | stmt 
identifier - 'i' | 'y | | '1' | | ' n ' 

Figure 2.6 shows stages in the top-down construction of a parse tree for 
die string: 

while n > n do a :» ■ - a 

Hie starting point is the root for the starting nonte rminal stmt . Initially, we 
are scanning the first, i.e., leftmost token of the input string, which is’ the 
taken while. The current token being scanned on the input in fnv p^ti y 
referred to as the lookahead symbol. From the lookahead symbol while it is 
evident that the while-production must be applied at the root. This deduction 
is possible because the while-production is the only production for stmt that 
can generate a string with while as its first symbol. Having on the 

while-production, the lookahead symbol while can be shifted, and the next 
input token read. 

At this point, suppose that an appropriate subtree for exp generating the 
substring »>n can somehow be constructed. Just after the subtree for exp is 
constructed, the lookahead symbol should be do or else the parser should 
decla r e that an error has been mnA» vv—— 

First Symbols Generated from a Nonterminal 

The lookahead symbol is a after do is shifted. There is no production for 
stmt that starts with the token ■. Sbce the right hand side of a production 
can start with a nonterminal, the lookahead token may be generated by such a 
nonterminal. In this grammar, the occurrence of identifier in the production 
stmt -» identifier 's» # exp 

implies that any token that can be the first symbol generated by identifier can 
also be the first symbol generated by stmt. 

Productions with e on the right hand side complicate the determination cf 
the first symbols generated by a nonterminal. It is not the case above but if 
identifier could derive the empty string, then the production 
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Fig. 2.6. Stages in the top-down construction of a parse tree. In a 
recursive-descent parser the construction of a nonleaf node 
corresponds to a procedure call; tokens at leaves are matched with 
input symbols. The lookahead symbol is used to decide which pro¬ 
duction to use. 

stmt — identifier 'exp 

would also be used if the lookahead symbol were 

More precisely, let a be the right hand side of a production far nontermi¬ 
nal A. We need to determine FTRSTT(a), which is the set of tokens that are 
the first symbols of some string generated from a. If a is e or can generate e, 
then e is also in FIRST(a). Far example, FIRSIXcptJtmtj) contains e. In 
practice, FIRST sets are easy to construct; an algorithm is given in Chapter 4. 

The FIRST sets are used if there is another production far A with right 
hand side 0. Recursive-descent pars ing requires FTRST(a) and FIRSr(0) to 
be disjoint Disjcintness of these sets implies that the lookahead symbol can 
be used to decide which production to use; if the lookahead symbol is in 
FIRST(o), then a is used. Similar remarks apply to 0. 
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When to Uae e Productions 

The final detail concerns the use of e-productions. We shall use c-productions 
as a default, i.e., if no other production can be used. If the lookahead symbol 
does not belong to the context, then an error will soon be detected. For 
example, consider: 

strni - 'begin' opLstmts 'end' 

optstmts - stmJist | e 

While constructing a subtree for optstmts, if the lookahead symbol is not in 
FIRST(stmJist) then the e-production is used. The check that any lookahead 
symbol other than end results in an error will take place as part of the con¬ 
struction of the subtree for stmt . 


Recnnfme-Deacent Paring 

A recursive-descent parser has a procedure for each nonterminal. The pro¬ 
cedure does two things. 

1. It decides which production to use by looking at die lookahead symbol. 
The production with right hand side a is used if the lookahead symbol is 
in FTRST(a). A production with e on the right hand side is used if the 
lookahead symbol is not in the FIRST set for any other right hand side. 

2. The procedure applies a production by mimicking the right hand side* a 
nonterminal results in a call to the procedure for the nonterminal, and a 
token matching the lookahead symbol results in the next input trfr-n bring 

read. If the token in the production does not match the lookahead symbol 

an error is declared. The process of matching a token t with the look* 
head symbol and reading the next input symbol if die match ^ 

be called shifting the taken t. 

For example, the production 

stmt - 'while' exp 'do' stmt 
leads to the code: 

shift 'while'; 
call procedure for exp; 
shift 'do'; 
call procedure for stmt ; 

In summary, a grammar i s suitable for recursive-descent parsing if the 
FIRST sets of all productions for a nonterminal are 
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A Pan** for Lists of Digits 

The discussion of recursive-descent parsing will be made more concrete by 
writing a parser for lists of digits separated by minus signs. There are two 
reasons for considering lists. First, the grammar is small enough that all 
details <>' die parser can be shown. Second, the grammar for lists in Example 
2.1 cannot be used directly, so the process of adapting a grammar for 
recursive-descent parsing can also be illustrated. By way of perspective, the 
techniques needed to adapt the gramm ar far lists suffice to construct a suitable 
grammar for all of Pascal. 

We shall use the programming language C to express our parser for lists. 
For those unfamiliar with C, we shall mention die salient differences between 
C and other Algol derivatives such as Pascal, as we find uses for those 
features of C A program in C consists of a sequence of functions, with exe¬ 
cution starting at a distinguished function called main. Functions cannot be 
Functions communicate either by pa*dng parameters by value or by 
accessing data global to all functions. Since the lookahead symbol needs to be 
consulted by the procedure far each nonterminal, we declare lookahead to be 
global by declaring it outside any function body. 

The meanings of some of the operators in C are as follows: 

■ assignment 

■■ equality test 

I ■ inequality test 

Sinm tokens are individual char a c te r * for our list-cf-digits example, sup¬ 
pose successive tokens are supplied by a function getchcr. However, looka¬ 
head is declared to be an integer to allow far additional tokens dial are not 
single characters, and are represented by integers greater than 2S5. Shifting 
of tokens will be performed by a function shift that reads the next input token 
if the lookahead symbol is matched, calling an error routine if not 

tnt lookahead ; /* global data •/ 

void shift It) 
tnt f; 

{ 

tf ( t !■ lookahead ) 
error (); 

else 

lookahead ■ getchar (); 

return; 

) 

The keyword void indicates that the “function” shift is really a procedure; 
i.e., it has no return value. Parentheses enclosing function parameter lists are 
needed even if there are no parameters; note error () and getchar () in the 
above program fragment. Also note that in C, if-statements do not use the 
keyword “then” and parentheses around die condition are required. The 
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else-portion of an if-statement is preceded by a serrrieden. 

Since the right hand sides of the digit productions consist of single t okens 
each production is handled by simply shifting the erprtrd tnkwn 

void digit () 

{ 

If ( lookahead ■■ '0' ) 

shtftr O'); 

eiae If ( lookahead ■» *y ) 

shtftrV)', 

dae if ( lookahead ■» '9' ) 
sh$C 9 '); 

dse 

error (); 


} 


Parser Termination 

It is possible for a recursive-descent parser to loop forever. The problem 
arises with productions like 

list - list digit 

Suppose the procedure for list decides to apply this production. The right 
hand side begins with list so the procedure is called recursively. But note that 
the lookahead token changes only when a terminal in the right hand side is 
shifted. Since the production begins with list, no changes to the input take 
place between recursive calls and the parser will loop. 

Fortunately, it is easy to characterize the grammars on which recursive- 
descent parsers loop; they are exactly the left-recursive gr amm ars in 

Chapter 4. The systematic elimination of left recursion is a generalization of 
the following modification to the gr ammar far lists of digits. 

EUmlnatioo of Left Recardao 

Recursive-descent parsing of lists of digits wfll be based on a new grammar 
constructed by replacing the nff«*ting production: 

list - list digit 

Repeated application of this production builds up a sequence of minus-digit 
pairs to the right of list. The new grammar generates minus-digit pairs dif¬ 
ferently. We use a new starting nonterminal rlist to avoid confusion. 
rlist — digit pairs 

pairs - digit pain | e 

The procedures for nonterminals will be collected into a complete C 
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program in the next section where translation of lists is considered. 

Figure 2.7 shows how the new gramm ar generates 2-3-5. The produc¬ 
tion 

pairs ■* digit pairs 

is right-recursive because this production far pairs has pairs itself as the last 
symbol on the right hand side. Right-recursive productions lead to trees that 
grow down towards the right, as in Fig. 2.7. 

Unfortunately, as was mentioned in Example 2.7, the minus operator is 
left associative, so a tree growing towards the right does not correspond to the 
normal usage far the minus symbol. In the next section, however, we shall 
see that the pr o per translation of lists into postfix notation can still be a tta i n ed 
by careful design of the translation scheme. 



Fig- 2.7. Parse tree generating 2-3-5 according to a gr amm ar 
suitable for recursive-descent parsing. 


2.5 Abstract Syntax 

A gr ammar that is suitable f or describing th e sy nta x of a language is not 
necessarily suitable for translating the language. Same of the many considera¬ 
tions that go into selecting a translation grammar for a language indude: ease 
of specification, availability of a convenient syntax analysis method, ease cf 
error recovery and reporting, and the intended meaning of the language con¬ 
structs. This section explores the distinction between concrete syntax, which 
specifies in detail what a program looks like, and abstract syntax in which 
m pn-firinl rtittinrtir>n< of farm, uni m p ortant for translation, are eliminated 

Rramr** 2.10. Abstract syntax captures file intended meaning of a construct 
while concrete syntax specifies how the construct is written dwra as a string. 
Far example, the program fragments 
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while ( m > n ) m • m - n\ 
while m > n do m :■ m - n 

(from C and Pascal, respectively) have the same abstract syntax but superfi¬ 
cially different concrete syntax. 

Semantically insignificant changes in concrete syntax are referred to as syn- 
tactic sugar. Syntactic sugar can be very important for readability, but has HT 
tie effect an translation. □ 

Example 2.11. The parse tree in Fig. 2.7 does not match the mtwvWt 
abstract syntax of the expression 2-3-5 shown in Fig. 2.8. The latter figure 
shows the normal abstract syntax for 2-3-5 because - conventionally associ¬ 
ates to the left, and the tree of Fig. 2.8 correctly shows 2-3 grouped first. 
The structures imposed by the grammars for list and liist are two alternative 
concrete syntaxes for the string. The correct abstract syntax seems easier to 
obtain if we use the left-recursive grammar for list, but the right-recursive 
grammar for rlist m a ke s recursive descent parsing possible, as we saw in the 
previous section. □ 



2 3 


Fig. 2.8. Abstract syntax for 2-3-5. 


L-valnea and R-vahies 

There is a distinction between the meaning of identifiers on the left and right 
hand sides of an assignment, but it is customary to use the san* concrete syn¬ 
tax for expressions on either side, leaving the distinction to the abstract syn¬ 
tax. In each of the assignments 

i : ■ 5; 
i i ♦ 1; 

the right hand side specifies an integer value, while the left hand side specifies 
where the value is to be stored. Similarly, if p and q are pointers to charac¬ 
ters, then in 

pt q t; 

the right hand side q t specifies a character, while p t specifies where the char¬ 
acter is to be stored. 

The terms l-value and r-value refer to values that are appropriate on the 
left and right hand sides of an assignment, respectively. 
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Postfix Form for Lists of Digits 

When file concrete syntax is not dose to the abstract syntax, it may be very 
difficult to specify translations in terms of file concrete syntax. 

F.T«mpl* 2.12. A minor variant of the grammar for lists of digits from the 
last section is unsuitable for translation, even though it can be used for 
recursive-descent parsing. The grammar from the last section is: 

rlist -* digit pairs 

pairs - digit pairs | c 

Suppose we change the productions for pairs to: 
pairs — rlist \ t 

This change has little effect on parsing since it merely delays the construction 
of the vertices for digit and pairs by inserting an extra node for rlist. 

However, this grammar cannot be used to translate lists of digits into post¬ 
fix form. The key question in the translation into postfix form concerns the 
placement of the minus signs. In a translation based on the production 

pairs - rlist 

the minus sign can appear either before the translation of rlist or after. If it 
appears before, then 2-3-5 incorrectly remains 2-3-5 in translation; if 
after, then 2-3-5 is translated incorrectly into 235—. □ 

Semantic rules for translating lists of digits are: 

rlist - digit pairs { f[rfof] - t[digit] t[pairs] } 

pairs 0 -* digit pairs x { t[pairs 0 ] m t[digit] t[pairs J ) 

pairs - c { t[pairs] ■ e } 

Figure 2.9 shows how 2-3-5 is translated using the above rules. Since 
pairs generates partial expressions, its translation is also a partial expression. 

For completeness we show how translation into postfix form can be dene 
during recursive-descent parsing. Before writing code in C we specify the 
translation semantic actions: 

rlist - digit pairs 

pairs - digit { print } pairs 

pairs — e 

digit - '0' { print '0' } 

digit - *9* { print '9' } 

C code corresponding to nonterminals rlist and pairs is shown in Fig. 

2 . 10 . 
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rlist {t ■ 23-5-} 


digit U = 2) pairs it = 3-5-} 

2 - digit {f - 3} pairs it - 5-} 

3 - digit {f - 5} pairs {f - e} 

I I 

5 e 


Flf. 2.9. The transJatian of 2-3-5 shown above can be imple¬ 
mented during recursive-descent parsing. 


void rlist C ) 

{ 

digit i); pairs (); 

} 

void pairs (} 
i 

If ( lookahead 

shiftidigit (); printfl m - m ) ; pairs (); 

} 

else ; /• no action because of e •/ 

) 

Fig. 2.10. Functions for the nonterminals rlist and pairs. 

2.6 Translation of Expreskms 

In this section we show how the concrete syntax for expressions can be sys¬ 
tematically constructed from information about the associativity and pre¬ 
cedence of operators. The construction views expressions as lists of lists; each 
list will be implemented like the translator in Fig. 2.10. 

Associativity and Precedence 

By convention, 2-3-5 is equivalent to (2-3J-5 while 2+3*5 is equivalent 
to 2+(3*5). When a potential operand like 3 has operators to its left and 
right, conventions are needed for deciding which operator take* that 

We say that operator - associates to the left Hwniw an operand with - 
on both sides of it becomes an operand of the - operator to its left. This 
definition is consistent with the discussion of lists in Section 2.4. The 
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definition extends naturally to other operators and to operators that associate 
to the right. 

Associativity alone is not enough to eliminate ambiguity when more than 
one operates is involved. Even if both + and • associate to the left, Fig. 
2.11 shows the two possible ways of supplying the operand 3 to an operator 
in 2*3*5. 


• 

/ \ 

+ 5 

/ \ 

2 3 

00 



2 • 


/ \ 

3 5 

(b) 


Fig. 2.11. In the expression 2*3*5, 3 can either be (a) die right 
operand of *, or (b) the left operand of *. The c or rec t choice is 
(b). 


We say that • has higher precedence than * because • takes its operands 
before * does. For example, 3 is taken by * in both 2*3*5 and 2*3*5; 
i.e., the expressions are equivalent to 2+ (3*5) and (2*3) *5, respectively. 
Precedence relations between other pairs of operators can be defined similarly. 

The abstract syntax of expressions can be specified by a table showing die 
associativity and precedence of operators within expressions. The following 
table shows the operators of Pascal in order of increasing precedence. With 
the exception of not, all operators are binary and associate to the left More 
precis^y, operators on a given line have the same precedence; an operator on 
a given line has lower precedence than an operator on the next line. 

<<■■<>»■> in 

♦ - or 

• / div nod and 

not 

A similar table for C has thirteen precedence levels. 

Associativity and precedence are related properties. Let subscripts / and r 
(from left and right) distinguish the two instances of - in 2 3 - r 5. The 

left associativity of - can be expressed by saying the -/ on the left takes its 
arguments before i.e. has higher precedence than Saying that all 
operators on the same line have the same precedence therefore farces all 
operators to associate in the same direction. 
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Concrete Syntax for Expressions 

A grammar for expressions will be constructed systematically from a table 
showing die associativity and precedence of operators. As an extended exam¬ 
ple, we start with the fair common arithmetic operators and the conventional 
precedence table (in order of increasing precedence): 

left associative: + - 

left associative: • / 


We creat e two nonterminals exp and term for the two levels ctf precedence, 
and an extra nonterminal factor for generating basic units in er prnwinn* The 
basic units in expressions are presently digits and parenthesized expressions. 
factor -digit | '(' exp •)• 

Now consider the binary operators, * and /, that have the highest pre¬ 
cedence. Since these operators associate to the left, the productions are simi¬ 
lar to those for lists that associate to the left. Lists of factors separated either 
by • or / signs can be generated using an auxiliary nonterminal multop : 

term - term multop factor | factor 
multop - | ’/• 

It is important to have the production generating factor from term ; otherwise 
there will be no way to generate digits or parenthesized expressions from 
term. 

The auxiliary nonterminal multop is avoided if the following productions 
are used. Both approaches specify the same language. 

term — term factor 
I term */* factor 
j factor 

Similarly, exp generates lists of terms separated by the additive operators. 
exp - exp ' + ' term 

I exp term 

j term 


Translation at Expressions 

"Die translation of expressions into postfix form is just like the translation of 
lists of digits into postfix form. As with lists, translations based on left- 
recursive productions are easy; the operator in the production is simply printed 
at the end of the production, as in 

exp - exp '+' term { print } 

A modified grammar suitable for recursive-descent parsing is obtained bv 
following the approach in Section 2.5: 
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exp term moreterms 

moreterms •* term { print '+' } moreterms 

| term { print } moreterms 

I « 

term - factor morefactors 

morefactors 

- *•' factor { print } morefactors 

| '/' factor { print 7' } morefactors 

I ‘ 

factor •* digit 

| '(' exp ')' 

The code far a recursive-descent parser can practically be read off the produc¬ 
tions. Before writing any code, we take a closer look at lists. 

Optimizing the Translator for Lists 

The resemblance be tween expressions and lists makes it possible to implement 
a translator from expressions into postfix notation using the implementation in 
Fig. 2.10 as a guide. The difference is that there are four levels of lists in the 
grammar for Pascal *rprr**inm The thirtwn precedence levels in C lead to 
thirteen levels of lists. 

Any improvements that can be in the translator for lists will pay off 
at each precedence level. We show that a single procedure suffices at each 
precedence level (instead of two procedures for the pair of nonterminals exp 
and moreterms and so on). 

For darity, the procedures rlist and pairs from Fig. 2.10 are shown bdow. 

void rlist l) 

{ 

digit (); pairs l)i 

} 

void pars () 

{ 

If ( lookahead — ) { 

shift digit ()S printf pairs (); 

} 

else ; 

) 

The recursive call to pairs in the procedure pairs is called tail recursive 
hrmnv the procedure returns immediately on return from the recursive call. 
Tail recursion can be replaced by iteration; simply replace the recursive call by 
a jump to the beginning of the procedure. Rewriting the code for pairs we 
g«: 

void pairs () 
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{ 

Siam If ( lookahead ■■ ) { 

sA# digit {); printf (•-■); goto Start ; 

) 

ebe ; 

} 

Note that pairs has turned into an iteration through the part of the list fol¬ 
lowing the fust digit; it looks for minus digit pairs. Since the only call of 
pairs is now from rlist, the two procedures can be integrated. 

The semantics of for-loops in C are such dial 

for ( ; ; ) 

repeats the following statement forever. Exit from foe loop occurs if and 
when the break statement is executed: 

void rlist () 

{ 

digit (); 
for ( ; ; ) 

If ( lookahead ) { 

shift digit 1); printf (■-■); 

} 


> 

Code far Translating Expressions 

Since exp and term have similar productions, we show the procedure imple¬ 
menting exp and moreterms only. 

void expl) 

{ 

term (); 
for ( ; ; ) 

If ( lookahead ■■'+'){ 

shft (' + '); term{); printf (■+•); 

} 

ebe If ( lookahead ■■ { 

jhjff term l) i printf (•-•); 

ebe 


> 

The code for a factor is straightforward; we check for either a 
parenthesized expression or a digit 
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void factor () 
i 

If ( lookahead ■■ ' ( * ) { 

sh$ri')i exp (); shjftl')'); 

) 

dae 

digit (); 

> 

Since there are ten possibilities for the first symbol generated by digit the 
call to digit is made a default in the code for factor above. Technically, we 
should test and see if the lookahead symbol is in FIRSr(di;if) before calling 
digit. No harm is done because a lookahead symbol outside HRST(<£;if) will 
lead to an error in the procedure for digit. 

The approach in the next section systematically treats all integer constants 
together. 

2.7 Lexical Analysis 

As shown in Fig. 2.12, a lexical analyzer reads the input to the compiler a 
character at a time and partitions the input into a stream of tokens. The 
parser then attempts to put a grammatical structure on this stream of tokens. 
Recall that a context-free gr ammar Aefim*. a lan g ua g e, to be a set of strings of 
tokens. So far in this chapter, tokens have been individual characters. In 
general, however, a single token may be a string of characters of arbitrary 
length. In this section we shall show how a lexical analyzer can insulate a 
translator from the actual representation of tokens. To begin, we mention 
some of the central functions of a lexical analyzer. 



Fig. 2.12. A lexical analyzer converts a stream of characters into 
a stream of tokens. 
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Removal of White Space and Comments 

Ibe expression translator in the last section looks at every character in the 
“put, so extraneous characters like blanks will cause it to fail Many 
languages allow “white space” (blanks, tabs, and newlines) to appear between 
any two tok«js. Comments are likewise ignored by the parser and translator, 
so commas may also be treated as white space. Modifying the grammar to 
allow white spate is not nearly as easy as inserting a lexical analyzer between 
the input and the parser. 


Anytime a single digit appears in an expression it seems reasonable to allow 
an arbitrary integer constant in its place. Since an integer constant is a 
sequence of digits, integer constants can be allowed either by produc¬ 
tions to the gra mmar for expressions, or by creating a token for such con¬ 
stants. The job of collecting digits into integers is generally given to a lexical 
analyzer because an integer forms a logical unit in expressions. 

Let number be a new token representing an integer. When a sequence of 
digits appears in the input stream, the lexical analyzer will return number for 
parsing purposes. The value of the integer will be passed along as an attribute 
of the token number. Logically, the lexical analyzer passes the attribute 
along with the token to the parser. Writing a tnkm and its attribute as a 
tuple enclosed between <>, the Input 

31 + 28 + 31 

results in the sequence of tuples: 

<NUMBER,31> <+,> <NUMBER,28> <♦,> <HUKBER,31> 

Altritxites play no role during parsing, but are needed during translation. 


Recognizing Identifiers and Keywords 

Many languages use identifiers as names of variables, arrays, functions, and 
the like. The main problem in having identifiers as tokens is that multiple 
uses of the same identifier have to be recognized. For example, it is signifi¬ 
cant that the same identifier appears on both sides of the assignment: 
count :■ count +1; 


Some mechanism for saving the characters of an identifier and looking up 
identifiers is therefore needed. A symbol table is often used as such a 
mechanism, although symbol tables typically contain additional information 
about identifiers as well, e.g., the type of the identifier (whether it is integer 
real, etc.) or the storage associated with it (static, dynamic, etc.). 

Many languages contain fixed character strings like begin, end, if, 
then, and so cm, that serve as pinctuation marks or to identify’certain coo- 


28 INTRCDUCTICN TO COMPILERS 


2.7 


satisfy the rules far farming identifiers, so a mechanism is needed for deciding 
when a sequence of characters is a keyword and when it is an identifier. The 
problem is easier to resolve if keywords are reserved ; i.e., if they cannot be 
used as identifiers. Then a character string can be treated like an identifier if 
it is not in a table of keywords. 

A related problem arises if the same characters appear in more than one 
token, as in the three tokens <, <■, and <> in Pascal. Techniques for isolat¬ 
ing such tokens efficiently are discussed in Chapter 3. 


Producing and Conanming Tokens 

The interaction between the lexical analyzer and parser can be implemented 
by setting up a producer-consumer relationship between them, as shown in Fig. 
2.12. The lexical analyzer produces tokens and places them in a token buffer, 
while the parser consumes the tokens in the buffer. The interaction between 
the two is constrained only by the size of the buffer because the lexical 
analyzer cannot proceed when the buffer is full and the parser cannot proceed 
when the buffer is empty. An interesting special case occurs when the buffer 
can hold just one token. In this case the interaction can be implemented sim¬ 
ply by making the lexical analyzer a procedure called by the parser, returning 

tnlrrrn on demand 

The input buffer cannot be implemented quite as simply. Far example, if 
the lexica] analyzer is collecting digits to form integer constants, it can stop 
only when a non-digit appears. But at fins point die lexical analyzer has con¬ 
sumed one character too many, so the character has to be returned to the 
input buffer. In same situations the lexical analyzer has to read several char¬ 
acters ahead. Input characters may also need to be saved for error reporting, 
since some indication has to be given of where in the input text the error 
occurred. 

A Lexical Analyzer 

We shall now construct a lexical analyzer that interacts with an input buffer 
through two routines called getchar and ungetchar , for getting and returning 
characters to the buffer, respectively. The lexical analyzer will be a procedure 
called gettoken, which can be called by the parser to produce tokens on 
demand. The expression translator of Section 2.6 obtained characters by cal¬ 
ling getchar. Our lexical analyzer can be invoked by calling gettoken instead 
of getchar. 

Consider the function gettoken in Figure 2.12. It skips over white space 
keeping track of line numbers for possible error messages. Note that gettoken 
returns integer encodings of tokens. For characters, the encoding will be the 
standard wwxting provided by the programming language C - an integer 
between 0 and 235. Additional tokens will be encoded as integers greater 
than 255. 
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hit tineno ; 


Int gettoken (); 
i 

t; 

to ( i ; ) { 

t • getchar (); 

if ( r ■■ BLAHJC II t ■■ TAB ) 
/* do nothing */ ; 
else if ( r newline ) 

tineno - lineno + 1; 


} 

} 


return t ; 


Fig. 2.13. A lexical analyzer that eliminates white space. 


Numbers Instead of Digits 

Allowing numbers in expressions instead of digits, requires a change in the 
grammar for expressions. The change to the gr ammar in Section 2 6 is 
minimal, since digit appeared only in a production for factor. The new pro- 
ductiom and semantic actions for factor are shown below. Note that nonter¬ 
minal <hgu has been replaced by a token number. Instead of printing the 
digit, the new action prints the value attribute of number: 

factor - '(' exp ')' 

I number { print v[number]; ) 

The key problem in implementing this translation is the communication of 
attribute values from the lexical analyzer to the procedure for factor The 

Cf F ‘«- 2-14 cm be used with a 

langu^e that does not allow data structures to be returned as results (cf Pas¬ 
cal). The lexical analyzer is a subroutine returning tokens. Attribute values 
are returned through a global variable tokenval. The lexical analyzer in Fig¬ 
ure 2.14 is obtained from that of Hg. 2.13 by adding code to collect digits into 
single number tokens. 

7116 tencal analyzer uses two procedures isdigitU) and digitvalU) to test 
if f is a digit and to determine its value as an integer, respectively. 

The procedure for factor is a straightforward implementation of the 
,bove ' N “ ot 

void factor () 
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tnt lineno ; 
but tokenval ; 

fait gettoken (); 

{ 

tnt t ; 

for ( ; ; ) { 

t - getchar (); 

If ( t »■ BLANK I I r »■ TAB ); 
dac If ( t ■■ HEMLINE ) 

lineno • lineno ♦ 1; 
che If ( isdigit (r) ) { 

tokenval • digitvaUt ); 
t ■ getchar l); 
while ( isdigit It) ) { 

tokenval ■ tokenval * 10 + digitval (r); 
t • getchar (); 

> 

in getchar It) i 

return NUMBER; 

) 

ebe { 

return t ; 
tokenval • NONE; 

} 

) 

} 


Fig. 2 .14. This lexical analyzer is constructed by adding code for 
collecting integer constants to the lexical analyzer in Fig. 2.13. 
Attribute values for token number are passed by setting a global 
variable tokenval . 


^61 4 . 

If ( 

) 


lookahead »■ ' (* ) { 

shift ('('); cqp(); shiftl')'); 

If ( lookahead NUMBER ) { 

printf (" Xi m ,tokenval )i shift (HUMBER); 


> 


error ("factor"); 
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46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 

]• I- I- 1° 1 ° l u l n l fc Igosli IbosI . 1.1.1 1 I 


String Table 


Entry 

Token 

Attribute 




11 

ID 

49 


ID 

55 





Symbol Table 


Fig. 2.15. Symbol table and array for storing strings. 


Identifier! and Symbol Tables 

Since instances of an identifier have to be reco gnized, die characters in the 
identifier must be saved. Typical of the operations needed to add and retrieve 
strings of characters to a symbol table are the following. 

insert: add a new string to the table 

lookup : search for a string 

lookadd : lookup and insert if not found 

prstring : print a string 

A particular implementation is sketched in Fig. 2.15. Some identifiers 
may be short and others quite long, so space for the characters in a string is 
taken as needed from a linear array. Each string is terminated by an end-of- 
string symbol denoted by EOS. 

Figure 2.15 shows two identifiers, count and i, occupying postians start¬ 
ing at 49 and 55, respectively, in the linear array constituting the string table. 
Retrieval is facilitated by keeping these starting positions as attributes in the 
symbol table. As shown, the 11th and 12th entries are for the two identifiers. 

The additional field for tokens in each entry in die symbol table is redun¬ 
dant at the moment, since the only new token is ID, far identifiers. How¬ 
ever, the token field leaves open the possibility of using the mnv» implementa¬ 
tion to keep trade of other strings, like keywords. 

The insert and lookup operations take a string as an argument and return 
the position of the symbol table entry for the string. The operation for 
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printing a string, on the other hand, takes the position of a symbol table entry 
and prints the associated string. 

The changes needed to allow identifiers in expressions are similar to those 
for allowing numbers instead of digits in expressions. This time a new pro¬ 
duction and semantic rule are added for nonterminal factor. The integer 
attribute p of token id gives the position of the symbol table entry for the 
identifier. 

factor -» '(' exp *)* 

| HUMBER { print v [bomber] ; } 
j ID { print identifier p[lD]; } 

As was the case with token number, die lexical analyzer has to be modi¬ 
fied to collect identifiers, save them, and to return attribute p through the glo¬ 
bal variable tokenval. The code that has to be added to gettoken in fig. 2.14 
is suggested by: 

ebe if ( isalphaU) ) { 

collect letters and digits in a buffer 

lookup (and if not found insert) the string 

set tokenval to the position of the symbol table entry 



Keywords and Composite Tokens 

The compiler in the next section is for a language with the expressions of Pas¬ 
cal. Several of the operators of Pascal are composed of letters, with mod and 
and being two examples. Fortunately, keywords are reserved in Pascal, so all 
instances of nod refer to the operator and cannot be confused with identif¬ 
iers. Thus, there is a token AND for the keyword and, a token NOD for the 
keyword tod, and so on. 

The implementation of reserved keywords is quite simple. Cbce a string 
of letters has been collected, we first check if die string is a keyword. If it is, 
a token for that keyword is returned; otherwise, the string is an identifier. 

The checking for reserved keywords can be done very simply by entering 
keywords into the symbol table along with the identifiers. The main differ¬ 
ence between identifier count and keyword sod is that token id (with an 
ap p r opriate attribute value) is returned for count, whereas token nod is 
returned for the keyword. If the token field of the keyword entry is initial¬ 
ized to MOD, thm the lexical analyzer can handle identifiers and keywords in 

the same manner Assuming that the initialiratinn ha* hi- e n done, the tT>d f 
for handling both identifiers and keywords is suggested by: 

ebe if ( isalphaU) ) { 

collect letters and digits into buffer ; 
p m lookup (buffer ); 

If ( found (p) ) { 
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tokenval ■ p ; 

return symbohabU Ip ] . token; 

) 

ehe { 

tokenval ■ insert (buffer , ID ) ; 
return id; 

) 

) 

Finally, we need to deal with composite tokens like <- and <>. Since the 
number of such tokens is small, it is easy to devise an ad-hoc test for 
A systematic approach is explored in a later chapter. 

Once the lexical analyzer is modified to return distinct tokens for eadi 
operator in Pascal, a recursive-descent parser can be constructed following the 
outline in Section 2.6. 


An abstract machine is an interface between a mnrhiTH^ indfp wwWit program¬ 
ming language and a language-independent target machine. The front end of 
a compiler determines if a program is well formed and what that farm is, 
while code for a target m ac h i n e is generated by the back end. It is con¬ 
venient, though not necessary, to use an abstract machine as the interface 
between the two ends. 

A well-defined interface makes the front end independent of the target 
machine, making it easier to move the compiler to a different machine. A 
quick (if inefficient) compiler can be constructed by interpretin g the abstract 
tn ftf ^ nc or by writing a simple-minded code generator. Bootstrappin g tech¬ 
niques discussed in Chapter 11 can then be applied to get an efficient imple¬ 
mentation. 

In this section we consider an abstract marhine and show how code can be 
generated for it. The machine has separate instruction and data memories and 
a stack for evaluating expressions. The instructions are quite limited and fall 
into three classes: arithmetic; stack manipulation; and control flow. Fig. 2.16 
illustrates the ma c h i ne . The pointer pc indicates the instruction we are about 
to execute. The meanings of the instructions shown will be discussed shortly. 

Arithmetic Instructions 

The machine implements the expressions of Pascal. Basic operations like 
addition and subtraction are supported directly by the abstract machine. In 
fact, since Pascal does not have too many operators, it makes sense to inri yrif 
an instruction for each operator.f The back end will then be able to choose 
the target code that best i mp le m e nt s each operator on the target machine. 

+ The abstract machine used in the Pascal P compiler actually has insmicticra fcr both in¬ 
teger and real versions of each arithmetic operator. 
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A Stack Machine 



Instructions 

Stack An* 

Data 

1 

push 5 


7 


0 

2 

rvalue 2 


16 


11 

3 

+ 


1 


7 

4 

rvalue 3 



5 

• 

-pc 



6 





Fig. 2.16. Snapshot of the machine after the first four instructions 
are executed. 

The boolean operators and and or require both their arguments to be 
evaluated; an alternative is discussed along with conditional statements. 

F*rh arithmetic instruction pops its arguments off the evaluation stack and 
pushes its result onto the stack. The value on top of the stack is taken to be 
the right operand, so 5-3 is computed by pushing 5, pushing 3, and sub¬ 
tracting. All values are integers, with 0 corresponding to false and 
nonzero positive integers corresponding to true. Rather than invent new 
names for operators we use die Pascal representations in writing stack 
machine code. Thus, the add instruction is ♦. 

Stack Manipulation 

BrsKir* the obvious instruction for pushing an integer constant onto the stack 
and popping a value from the top of the stack, there are instructions to access 
data memory. As in Section 2.5, the term r-value refers to an integer value 
that can appear on the right hand side of an ass i gnm e n t, while the term 1- 
value refers to the data location that can appear on the left hand side. 

The instructions are: 

push v onto the stack 
push contents of data location l 
push address of data location / 
throw away value an top of the stack 
the r-value on top is placed in tire 1-value below it 
and both are popped 
swap the top two values on the stack 


push v 
rvalue l 
lvalue / 
pop 


■wap 
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Translation at Expressions 

Code to evaluate an expression on a stack machine is essentially postfix nota¬ 
tion for that expression. Here we will generate assembly code in which data 
locations are addressed symbolically. The selection of data locations for iden¬ 
tifiers can be performed by an assembler phase following the compiler or by 
the back end of the compiler itself. Using symbolic addresses, the expression 
a+b translates into: 

rvalue a 
rvalue b 
♦ 

In wards: push the contents of the data locations for a and b onto the stack; 
then pop the top two values on the stack, add them, and push the result onto 
the stack. 

Example 2.13. The assignment 

day :■ (1461#yy) div 4 + (153*« + 2) div 5 + d 
translates into 

lvalue day 
push 1461 
rvalue yy 
• 

push 4 
div 

push 153 
rvalue ■ 

* 

push 2 
♦ 

push 5 
div 
+ 

rvalue d 


:■ n 

The rule for translating assignments is suggested by the above example: the 1- 
value of the identifier assigned to is pushed onto the stack; the expression is 
evaluated; its r-value is assigned to the identifier. 

The same rule can be expressed formally as follows. Attribute t of exp 
and stmt gives their translations. Attribute jfr of id gives the string 
representation of the identifier. 

stmt — id exp 

{ t[stmt] = * lvalue ' rtr[lD] t[exp ] ) 
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Control Flow 


Hie stack machine executes instructions in numerical sequence unless told to 
do otherwise by a conditional or unconditional jump statement Several 
optics® exist for specifying the targets of jumps: 

1. The instruction operand gives the target location. 

2. The instruction operand specifies the relative distance, positive or negative, 
to be jumped. 

3. The target is specified symbolically; i.e., the machine supports labels. 

With the first two options there is the additional possibility of taking the 
operand from the top of the stack. 

We choose the third option for the abstract machine because it is easier to 
generate such jumps. Moreover, symbolic addresses need not be changed if, 
after we generate code for the abstract machine, we make certain improve¬ 
ments in the code that result in the insertion or deletion of instructions. 

The control flow instructions for the stack machine are: 


label / 
goto / 
gofalae / 
gotrue / 
halt 


target of jumps to /; has no other effect 
next instruction is taken from label l 
pop the top value; if it is zero jump 
jump if greater than zero 
stop execution 


Trandatkm of statements 

Conditional execution and iteration are implemented using labels and goto 
statements, so we concentrate on the placement of labels. 

The layout in Fig. 2.17 sketches an implementation of if- and while- state¬ 
ments. All we have to do to convert the layout into a syntax directed transla¬ 
tion is to show how appropriate labels can be generated for each if and while. 
Qearly all conditional and while statements cannot exit to the same label 
out. 

In the implementation below, a procedure newlabel returns a fresh label 
every time it is called. To use newlabel properly, we need to rewrite the con¬ 
crete syntax for statements. In particular we use a new nonterminal testgo, 
which recognizes an exp-, its translation has a jump added at the end of the 
translation of exp. testgo therefore has two attributes: t for its translation, 
nnH dest for the destination label of the jump. 

stmt — 'if' testgo 'then' stmt x 

{ t[stmt] “ t[testgo] fjjfm*!] 'label' dest[testgo ] > 
testgo - exp 

{ dest [testgo] ■ newlabel C) 

t[testgo] *= t[exp] 'gofalae' dest [testgo] } 
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label teat 

code for exp 


code for exp 

gofalae out 


gofalae out 

code for stmt! 


code for stmt! 

label out 


goto teat 

i 

label out 


Fig. 2.17. Code layout for if- and wMe-statarcnts. Tmtinm of 
the labels out and test suggest the flow of control in both lay¬ 
outs, and need to be replaced by appropriate unique labels during 
translation. 

As a first step towards converting the above specification into code, let us 
use actions to emit code. Instead of printing instructions directly, a procedure 
emu is used to hide printing details. For example, emit can worry about 
whether each abstract machine instruction has to be on a separate line. 
stmt — 'if' test go 

1 out :• dest [testgo]', ) 

'then' stmt! 

{ emit l 'label' ,out); } 

Note that calls to emit take care of the translation attribute t, but the dest 
attrilxite of testgo is returned explicitly, just like the attributes of numbers and 
identifiers in Section 2.7. In the following code, the dest attribute is returned 
by the procedure for testgo . Snce labels generated by the compiler will be 
11, 12, ■ • • a label can be recovered from its integer portion. Attribute 
dest is therefore implemented as an integer: 

int testgo () 

{ 

fat out ; /* integer portion of exit label */ 

exp (); 

out m nevAobel (); 
emit ( GOFALSE , out ); 
return out’, 

} 

void stmt () 

{ 
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tot test , ota ; /* for labels •/ 

if ( lookahead ■» ID ) { 

/• process an assignment •/ 

} 

dw If ( lookahead ■■ IP ) { 

shift ( IF) ; out m testgo (); stmt{)\ emit ( LABEL , out ); 

} 

/* code far remaining statements »/ 
die error ("stateaent expected near line”); 

) 

The code layout far while-statements in Fig. 2.17 can be converted into 
code in a similar fashion. The translation of sequences of statements is simply 
the concatenation of the statements in the sequence, and is left to die reader. 

Although the details will obviously be different, most single-entry-single¬ 
exit constructs can be translated as suggested above. We illustrate by consid¬ 
ering control flow in expressions. 

Example 2.14. The lexical analyzer in Section 2.7 contains a conditional of 
the form: 

If t - BLANK or t ■ TAB then • • • 

If t is a blank, then dearly it is not necessary to test if t is a tab, because 
either equality implies that the condition is true. The expression 

exp x or exp 2 

can therefore be implemented as suggested by 
if exfr then true else expi 

The reader can verify that the following layout implements the or opera¬ 
tor: 


code for exp x 

copy /* copy value of exp^ */ 

gotrue out 

pop /* pop value of exp x */ 

code for expj 
label out 


Recall that the gotrue and gofalse instructions pop die value on top of 
the stack to simplify code generation for conditional and while statements. 
Copying of the value of expi ensures that the value on top of the stack is true 
if die gotrue instruction leads to a jump. □ 
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Inheriting Context 

The procedure nevAabel in the translation of statements keeps track of the 
labels that have been generated Generated labels are the only information 
about context that is used in the translation of statements. Typical of other 
contextual information are declared types and the storage allocated for identif¬ 
iers. 

Compilers maintain contextual information in global data structures that 
are modified and accessed as the input is analyzed Finer control over the 
transmission of contextual information is possible using inherited attributes, 
discussed in Chapter 5. 

2.9 Putting the Compiler Together 

In this section we shall recapitulate the techniques presented in this chapter. 
In the process we construct a syntax-directed translator, in the form of a com¬ 
plete C program, from input expressions into output code for a stack machine. 
To keep the program manageably small we consider expressions consisting of 
lists of numbers separated by minus signs. The exercises at the end of this 
chapter suggest how the translator can be extended to handle some of the 
other language constructs in the chapter. 

The Translation to be Done 

We begin with a context-free grammar for our language: 

list — list factor | factor 

factor — NUMBER 

The tokens for this grammar are the mi mu sign ami number. Associated 
with token number are two attributes v and t , giving the value and the string 
representation of the number. (Note that attribute t can be reconstructed 
from the value.) 

The translator is expected to implement the following syntax-directed 
translation scheme: 

list 0 •* listi factor 

{ t[list 0 ] - t[listi] t[factor] } 
list — factor 

{ f[/irr] ■ t[factor] } 

factor -» NUMBER 

{ t[list] - t [NUMBER] > 

Since we shall construct our translator around a recursive-descent parser, 
the above left-recursive grammar for lists has to be transformed using the 
techniques of Section 2.4. We daim (without proof since techniques far 
showing equivalence have not been discussed) that the following gr ammar and 
semantic actions implement the above syntax-directed translation scheme. 
Support for the daim may be found in Section 2.5, where a simil ar gr ammar 
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appsW?, 


rlist 

— factor pairs 

pairs 

— factor 

pairs 

-* e 

factor 

-* NUMBER { 


{ emit } pairs 
emit r [number] } 


These semantic actions can be transformed into a working C program by 
using the techniques of Section 2.5. (We also a need a lexical analyzer to sup¬ 
ply taken number, but we will get to that in a moment.) If the techniques of 
Section 2.5 are used to transliterate the above semantic frinn* into code we 
get essentially the procedures for rlist and pain in Fig. 2.10. However, the 
implementation below follows the optimized translator for lists in Section 2.6; 
it folds the code for pairs into that of rlist. Additional arithmetic operators 
can be added as suggested in the same section. 


Doing the Translation 

The code for the compiler in this section has four parts: 

1. A lexical analyzer taken from Fig. 2.14. 

2. A parser for lists like the one in Section 2.6. 

3. A trivial emitter of abstract marhiTv code. 

4. Auxiliary code to knit the pieces together. 

The auxiliary code appears first in the working C program below. The 
first couple of lines, beginning with #inclode, define the predicate iadi- 
git and the file atderr for error messages. The token HUMBER is imple¬ 
mented as integer 256 by the line beginning with #define. A Pascal pro¬ 
gram would declare number in a const declaration. Functions that are called 
before they are defined are listed on lines beginning with extern. 

Minimal tailoring of error messages is performed by passing a string as a 
parameter to the function error. The error routine indicates the line 
number in the input where the error is discovered and prints out the next 
token. As the number of tokens increase, it would be worthwhile to set up a 
separate function to indicate the context of the error. As shown, all errors are 
fatal: the call exit (1) terminates execution. 

The main routine initialize* the lookahead token by calling the lexical 
analyzer through advance. It then initiates parsing by calling rlist, the 
function for the start symbol of the grammar. The end of the inpit is sig¬ 
nalled by a semi-colon, so aiain tests for a semi-colon when parsing is over. 
The compiler can be made easier to test by modifying sain to accept a 
sequence of expressions terminated by semi-colons. 

The lexical analyzer is accessed through advance, but the work is done 
in the function gettoken from Fig. 2.14. Function advance resets the 
global variable tokenval to a negative number, and then obtains the next 
token. This implementation differs from that earlier in this chapter where a 
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function shift checked that the token being shifted matched the lookahead 
character. The use of advance eliminates redundant comparisons. Stw. the 
characters for digits have consecutive integer representations, the integer value 
of a digit is obtained simply by subtracting the representation of zero. This 
subtraction constitutes the body of the function digitval. 

The emitter consists of two trivial routines; using separate mu tiny facili¬ 
tates additions to the language being c ompiled 
Try executing this program. 


/* 

Auxiliar y Code: global declarations and some functions 

♦/ 

^include <ctype.h> /* defines predicate iadigit •/ 

#include <atdio.h> /• defines error message file stderr •/ 

#deflne NUMBER 256 

int lookahead; /• global data */ 

int tokenval; 

int llneno ■ 1; 

extern void advanced, rliatd, factord; 

extern void emitld, e»it2(); 
void error(m) 

char «m; 

{ 

fprintf(atderr, "near line Xd: Xe", lineno.m); 
if ( lookahead < HUMBER ) 

fprintf(atderr, ", lookahead Xc\n", lookahead); 

else 

fprintf(atderr, a , lookahead Xd\n", tokenval); 

exit(1); /* stops execution •/ 

> 

void maind 

( 

advance (); /• initialization •/ 

rliatd; /* start parsing */ 

if ( lookahead t ■ ';' ) /* ';' denotes end of input */ 
error("expecting ;•); 

) 

/* 

Lexical Analyzer: advance () is the interface 

*/ 

int digitval(t) 
int t; 
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return t - 'O'; 

) 

ini gefctoken() 

{ 

infc t; 

( ; ; ) 
i 

t » getcharO; 

if ( t »» ' ' 11 t ■■ '\t' ); 
else if ( t ■■ '\n' ) 
lineno ■ lineno + 1; 
else if ( isdigit(t) ) 

< 

tokenval ■ digitval(t); 

while ( isdigit( t*getchar() ) ) 

tokenval ■ tokenval*10 * digitval(t); 
unge tc(t,stdin); 
return HUMBER; 

} 

else 

return t; 

) 

} 

void advance() 

{ 

tokenval ■ -1; lookahead ■ gettokenO; 

> 

/* 

Parser 

*/ 

void rlistO 

i 

factor(); 
for ( ; ; ) 

if ( lookahead ■» '-' ) { 

advance(); factorO; emitl 

} 

else 

break; 

> 

void factorO 

< 
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if ( lookahead -■ NUMBER ) { 
emit2( NUMBER ,tokenval); 
advance(); 

> 

else 

error("expecting number"); 

/* 

Emitter: emits stack nwrfiinp code 

*/ 

void emitl(t) 

int t; 

{ 

if ( t -» '-' ) 
printf("-\n"); 

else 

error("compiler error in emitl"); 

void emit2(t,val) 

int t, val; 

{ 

if ( t »■ NUMBER ) 

printf("push\tXd\n*,val); 

else 

error("compiler error in emit2"); 


Fig. 2.18. A simple compiler. 


2.10 EXERCISES 

2.1 Consider the context-free gr ammar 

s - SS ' + ' I SS I 'a' 

a) Show how the string aa+a* can be generated by this grammar. 

b) Construct a parse tree for this string. 

c) What language is generated by this grammar? Justify your 
answer. 

2.2 What language is generated by the following gr ammar s? In enrfr mv 
justify your answer. 

a) S - 'O' S '1' | 'O' ' 1 ' 

b) S - ' + '5 5 | '-'5 5 | 'a' 
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c) S - S 'i'S ')' S | * 

d) S - 'a'S 'b'S ! 'b'S 'a'5 | e 

e) 5- V|SVS|n|S | '('S ')' 

2.3 Which of the grammars in Exercise 2.2 are ambiguous? 

2.4 Construct unambiguous context-free gramm ars for each of the follow¬ 
ing languages. In each case try to show that your grammar is correct. 

a) Arithmetic expressions in postfix notation. 

b) Left-associative lists of identifiers separated by commas. 

c) Right-associative lists of identifiers separated by commas. 

d) Arithmetic expressions of integers and variables with the four 

binary operators ♦, *, /. 

e) Add to the arithmetic oper a t or s of (d) unary plus and minus. 

2J This exercise considers some of the expression operators of C In 
^arti of the following cases, construct an unambiguous grammar for 
expressions containing identifiers, integers, parenthesized expressions 
and the operators indicated. The operators are listed in order of 
decreasing precedence. 

a) binary operators • and ■, with • being left associative and ■ 
being right associative; 

b) to the operators of part (a) add a unary operator ++ that can be 
used as either a prefix or postfix operator and has higher pre¬ 
cedence than *; 

c) to the operators of part (b) add a conditional construct 
exp ? exp : exp , where the construction ? exp : is treated as a left 
associative binary operator with precedence between * and ■; 

d) starting with the operators of (c) allow * to be used as unary pre¬ 
fix operator with the same precedence as +♦, in addition to its 
role as a binary operator. 

2.6 a) Show that all binary strings generated by the following grammar 

have values divisible by 3. Use induction on the number of nodes 
in a parse tree. 

b) Does the grammar generate all binary strings with values divisible 
by 3? 

num ■* | # 1 # * 0' '0‘ '1' | num '0' | num rum 

2.7 Construct a context-free gramm ar for Roman numerals. 

2.8 Construct a syntax-directed translation scheme that translates arith¬ 
metic expressions from infix notation into prefix notation. Gve an 
example of an annotated perse tree for your translation scheme. 

2.9 Construct a syntax-directed translation scheme that translates arith¬ 
metic expressions from postfix notation into infix notation. Gve an 
example of an annotated parse tree. 
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2.10 Construct a syntax-directed translation scheme that translates arith¬ 
metic expressions in infix notation into arithmetic expressions in infix 
notation having no redundant parentheses. Show the annotated parse 
tree for the input (((i + 2) * (3 * 4 )) ♦ 5). 

2.11 Gmstruct a syntax-directed translation scheme that translates integers 
into Roman numerals. 

2.12 Construct a syntax-directed translation scheme that translates Roman 
numerals into integers. 

2.13 Construct recursive-descent parsers for die grammars in Exercise 2.2 
(a), (b), and (c). 

2.14 Construct a syntax-directed translator that verifies that the 
parentheses in an input string are properly balanced. 

2.15 I m ple men t a translator from integers to Roman numerals based on the 
syntax-directed translation scheme developed in Exercise 111. 

2.16 ®) Construct a syntax-directed translation to translate into 

stack machine code arithmetic expressions generated by the follow¬ 
ing grammar: 

exp — exp term 

| exp term 

j term 

term — term factor 
term '/' factor 
j factor 

factor - '(' exp ')' | HUMBER | id 

b) Transform the underlying grammar into a form so that it can be 
parsed using a recursive-descent parser. 

c) Add to the grammar of part (b) semantic actions that implement 
the syntax-directed translation scheme of part (a). 

d) Extend the compiler of Fig. 2.18 to translate these expressions into 
stack ma c hin e code. The lexical analyzer should allow the 
number and id to be separated by white space. 

2.17 Construct a set of test expressions for your compiler for Exercise 2.16 
so that each production is used at least once in deriving some test 
expression. Construct a testing program that can be used as a general 
compiler testing tool. Use your testing program to run your compiler 
on these test expressions. How would you prove your compiler works 
correctly? 

2.18 Extend the compiler of Exercise 2.16 to translate into stack m«rhin> 
code statements generated by the following grammar: 

stmt — identifier exp 

| 'if' exp 'then' stmt 
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| 'while' exp 'do' stmt 

| 'begin' optstmts 'end' 

opt stmts - stmUist | c 

stmt-list - stmUist 's' stmt \ stmt 

2.19 Construct a set of test statements for your compiler of Exercise 2.18 
so that earh production is used at least once to generate some test 
statement. Use the testing progr a m of Exercise 2.17 to run your com¬ 
piler on these test expressions. 

2.20 In the programming language C the for-statement has the form: 

for ( expi ; expi ; expj ) stmt 

The first expression specifies initialization for the loop, which consists 
of the statement stmt. The second expression is a test made before 
rarfi iteration of the loop; the loop is exited when die expression 
hrrrgnr* 0. The third expression specifies an incrementation that is to 
be performed after each iteration. The meaning of the for-statement 
is similar to 

expx ; while ( expi ) { stmt exp^ ; } 

Construct a syntax-directed translation scheme to translate C for- 
statements into stack machine code. 

2.21 Consider the following for-statement: 

for i s- litep 10 - juntfl 10 * j do j :» j + 1 

Three semantic definitions can be given for this statement One pos¬ 
sible meaning is that the limit 10 * j and increment 10 - j are to 
be evaluated once before the loop, as in PL/L For ex amp le, if j - 5 
before the loop, we would run through the loop ten times and exit. A 
second, completely different, meaning would ensue if we are required 
to evaluate the limit and increment every time through the loop. For 
example, if j - 5 before the loop, the loop would never terminate. 
A third meaning is given by languages such as ALGOL. When the 
increment is negative, the test made for termination of the loop is 
j < 10 * j, rather than i > 10 * j. For each of these three semantic 
definitions construct a syntax-directed translation scheme to translate 
these for-loops into stack machine code. 

2.22 Consider the following gr ammar fragment for if-then- and if-then- 
else-statements: 

stmt - 'if' exp 'then' stmt 

| 'if' exp 'then' stmt 'else' stmt 
| other 

where other stands for the other statements in the language. 
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a) Show that this gr ammar is ambiguous. 

b) Construct an equivalent unambiguous gr a mmar that awyin^ 
each elie with the closest previous unmatched then. 

c) Construct a syntax-directed translation scheme based on this gram¬ 
mar to translate conditional statements into stack mnrhi™. code. 

2.11 BIBLIOGRAPHIC NOTES 

This introductory chapter touches on a number of subjects that are treated in 
more detail in the rest of the book. Pointers to the literature appear in the 
chapters containing further mater ial 

A number of Pascal compilers are derivatives of the compilers written at 
EIH, Zurich, by Wirth and his colleagues. Wirth [1971] is a report on an 
early implementation. I mp le m e nt ation notes distributed with the Pascal P 
confer were finally published as Nori et al. [1981]. Code generation details 
are given by Ammann [1977]. Pascal compilers are unusually well docu¬ 
mented: Barron [1981] is a collection of papers; Pemberton and Daniels [1982] 
is a commentary cm the code of the Pascal P4 compiler (code not included); 
Berry [1982] is devoted to Pascal S, which is a subset of Pascal A-kiWH by 
Wirth [1981] for student use. 

Context-free grammars were introduced by Chomsky [1956] as part of a 
study of natural languages. Their use to specify the syntax of programming 
languages arose independently. While working with a draft of AlgcJ 60, John 
Backus “realized that there was trouble about syntax description. It was obvi¬ 
ous that [Emil] Post’s productions were just the thing, and I hastily adapted 
them to that use (Wexelblat [1981], p.162).” The resulting notation is a vari- 
ant of context free grammars. The scholar Panini devised an equivalent syn¬ 
tactic notation to specify the rules of Sanskrit grammar between 400 B.C. and 
200 B.C. (Ingerman [1967]). 

The proposal that BNF, which began as an abbreviation of Backus Normal 
Farm, be read as Backus-Naur Farm, to recognize Naur’s contributions as edi¬ 
tor of the Algol 60 report, is contained in a letter by Knuth [1964]. 

Syntax-directed definitions have long been used informally in mathematics. 
Them application to programming languages came with the use of a grammar 
to structure the Algol 60 report. Shortly thereafter, Irons [1961] constructed a 
syntax-directed compiler. 

Conway [1963] describes a (nonrecursive) implementation of recursive- 
descent parsing using an explicit stack. Recursive-descent parsing has also 
been attributed to Lucas [1961]. ^ 

McCarthy [1963] advocated that the translation of a language be based on 
abstract syntax. Abstract syntax trees can be given a more “abstract”’ formu¬ 
lation using initial algebras (Goguen et al. [1977]). It is not really necessary 
that abstract syntax be tree-like - any convenient representation will do. But, 
if the abstract syntax is too far from the concrete syntax then it may be neces^ 
sary to explicitly specify the translation from concrete to abstract syntax See 
for example the translations in Lucas-Walk [1970] and Marcotty-Sayward 
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The relation between iterative and recursive programs has been die subject 
of murh study. McCarthy [1963] and van Wijngaarden [1966] independently 
show how iterative programs can be converted into recursive ones. Paterson 
and Hewitt [1971] show that there are recursive programs that cannot be 
implemented by iterations without using an explicit stack. Elimination of cer¬ 
tain recursions appears in McCarthy [1963] but tail recursion is not mentioned 
explicitly. An early reference for this transformation is Gill [1965]. 
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I. INTRODUCTION 

Being able to specify and match various patterns of strings and words is 
an essential part of computerized information processing activities such as 
text editing [26], data retrieval [36], bibliographic search [1], query process¬ 
ing [3], lexical analysis [27], and linguistic analysis [7], This paper examines 
three basic classes of string patterns that are particularly useful in these 
activities and analyzes some of the time-space tradeoffs inherent in searching 
for these classes of patterns. 

The three classes of patterns considered are (1) finite sets of strings, (2) 
regular expressions, and (3) regular expressions with back referencing. Effi¬ 
cient pattern matching algorithms for each of these classes are discussed. 
Several of these algorithms have the pleasing property of being both useful in 
practice and interesting in theory. It may appear that we have too many 
classes of patterns and too many algorithms but we will discuss some of the 
reasons why no single approach is ideal for all applications. 

There are several issues that must be faced in designing pattern match¬ 
ing software. First there is the question of what class of patterns the user 
should be capable of specifying and what notation is necessary to describe a 
given pattern. A nonprocedural description of a pattern, such as a regular 
expression, may often be easier for a nonprogrammer to specify than a pro¬ 
cedural description, such as a SNOBOL program [19]. In fact, several 
attempts have been made to clarify and axiomatize the description of 
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SNOBOL-like patterns [18, 34], However, there is a limit to how large a 
class of patterns can be specified simply in a nonprocedural manner. 

The difficulty of constructing an efficient recognizer from a pattern 
specification must also be examined. For some applications such as text edit¬ 
ing a restricted class of patterns from which an efficient recognizer can be 
constructed quickly is sufficient. For other applications such as textual 
search we may wish to allow more complex sets of patterns but we may then 
have to spend more time preprocessing the pattern to find an efficient recog¬ 
nizer for it. The problem is further complicated because some nonprocedural 
descriptions allow complex patterns to be described for which efficient recog¬ 
nizers do not exist or are hard to find. Finally the time-space tradeoffs in 
efficiently implementing the recognizer must be considered. 

In this paper efficient algorithms for finding various subsets and super¬ 
sets of regular expression patterns are of special interest. The simplest such 
question is finding a finite set of keywords in a text string, a problem for 
which several interesting and particularly effective algorithms have been 
recently found. Although regular expressions cannot describe all patterns 
that occur in information processing systems, it is shown that some modest 
generalizations of the string matching problem are NP-complete. 


II. PATTERN-MATCHING PROBLEMS 


For the applications considered here, a pattern is merely a set of 
strings. Each string in a pattern is said to be matched by the pattern. The 
input to a pattern-matching problem is a pair (p, x) where p is a specifica¬ 
tion of a pattern and x is an input string. There are many notations for 
specifying patterns, such as programs, grammars, and automata. For the 
applications considered here, various variants of the regular expression nota¬ 
tion will be used. 

The desired output of a pattern-matching algorithm also depends on 
the application. For example, if x is a sequence of lines of text (such as a 
dictionary, a program listing or a manuscript), then one useful output in edit¬ 
ing and searching applications is all lines containing a substring matched by 
p. Another possible output is the leftmost longest substring of each line that 
p matches. A third possible output is all substrings matched by p, but here 
we have to be careful when p denotes the empty string. For the purposes of 
this paper, however, we will just consider the output to be “yes” if x con¬ 
tains a substring denoted by p, “no” otherwise. 

We will measure the performance of a pattern-matching algorithm by 
the time and space taken to find a match measured as a function of the 
lengths of p and x. We will assume the pattern is given before the input 
string x. In this way an algorithm can construct from the pattern whatever 
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kind of pattern-finding machine it needs before scanning any of the text 
string. 


A. Regular Expressions 

Much of formal language theory deals with methods for specifying pat¬ 
terns as we have defined them. For the classes of applications listed above 
regular expressions provide a convenient and easy-to-use notation for describ¬ 
ing patterns. Classical automata theory defines a regular expression and the 
pattern it denotes as follows: 

(1) A character a by itself is a regular expression denoting the pattern {a}. ; 

(2) If r\ and r 2 are regular expressions denoting patterns p x and p 2 , respec¬ 
tively, then 

(a) rj | r 2 is a regular expression denoting p x JJ p 2 . 

(b) (n)( r 2 ) is a regular expression denoting the concatenation of p x 
and p 2 , that is, the set of strings {xy \ x is in p x and y is in p 2 }. 

(c) (/•])* is a regular expression denoting (Jp' where p‘ is the con¬ 
catenation of p with itself i times. By convention p° is the empty 
string, which we denote by e. 

(d) (/•]) is a regular expression denoting p x . 

Unnecessary parentheses in regular expressions can be avoided by 
adopting the convention that the Kleene closure operator * has the highest 
precedence, then concatenation, then |. All operators are left associative. 
For example, under this convention (a | ((6)*)(c)) can be written as 
a | b*c. 

Regular expressions can be used to specify patterns that arise in many 
diverse applications such as shortest path problems, program testing, printed 
circuit board layout, graphics, and programming language compilation [2, 4, 
22], Here are some simple examples of regular expressions from text-editing 
applications. 

(1) The regular expression ( apple\blueberry)-(pie\tart ) matches any of the 
four delicacies: apple-pie, apple-tart, blueberry-pie, blueberry-tart. 

(2) The regular expression the (very,)* very old man matches the strings 
the very old man; the very, very old man; the very, very, very old man; 
and so on. 

In several applications, it is convenient to embellish the definition of 
regular expressions with various notational shortcuts. One simple extension 
is to introduce metacharacters that have specialized meanings. For example, 
in the definition above the symbols |, *, (, and ) are metacharacters not 
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considered to be part of the pattern alphabet. To include these symbols as 
targets for matches, we can introduce another metacharacter, say \~ as an 
escape character. We can then use \| to denote the pattern { | }, \* to denote 
{*}, and so on. \\ denotes {\}. 

We can also introduce metacharacters to match various positions in a 
string. The metacharacters " and $ will denote the left and right ends of a 
string. Thus dous$ will match dous only at the right end of a string. 

One other convenient shorthand is a succinct notation for character 
classes. The symbol 2 will be used to denote the entire string alphabet. We 
can think of 2 as a “don’t care” symbol that matches any symbol. We will 
use the regular expression [ abc] to denote the set {a, b,c] and [- abc] to 
denote 2 - {a, b, c}. Thus the regular expression 

“[- aeiou ] *a[-aeiou]*e [- aeiou ]*/ [- aeiou]*o[ - aeiou ]*«[- aeiou ]* $ (1) 

will match all strings in which the five vowels appear in lexicographic order. 
For example, (1) matches the string “abstemious.” 


B. Regular Expressions with Back Referencing 

All the extensions of the regular expression formalism discussed so far 
have been notational shorthands. One useful extension that increases the 
expressive power of regular expressions is back referencing. This extension 
allows complex patterns that cannot be specified by regular expressions alone. 
The back referencing concept was introduced in the first version of SNOBOL 
[11] and has appeared in Dennis Ritchie’s implementation of the UNIXt 
pattern-matching program grep [25], 

Tlie following rules define the syntax of a regular expression with back 
referencing (rewbr for short). In these rules 2 is a finite alphabet and N = 
{vi, v 2 , • • • } is a set of variable names. 

(1) Each character a in 2 is a rewbr. 

(2) Each variable v, in A' a rewbr. 

(3) If rj and r 2 are rewbrs, then so are 

a) r j/| r 2 

b) (r,)(r 2 ) 

c) (r i)* 

d) (r i) 

e) rj.v, 


UNIX is 
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In rule 3(e), the dot is the back referencing operator. Its properties are simi¬ 
lar to those of the conditional value assignment operator in SNOBOL4 [191. 
As before, redundant parentheses can be avoided using the same precedences 
and associativities as with regular expressions. The back referencing operator 
is left associative and has the highest precedence of all operators. 

The pattern denoted by a rewbr is more difficult to define than the pat¬ 
tern denoted by a regular expression because each rewbr can contain vari¬ 
ables that may be assigned values by the back referencing operator. We shall 
define the pattern denoted by a rewbr r in terms of an intermediate quantity 
V(r), the value of r. V(r) will be a set of pairs of the form (s, a) where 5 is 
either null or a string in (2 (J V)* and a is an assignment of strings in 

(2 |J V)* to the variables in N. V(r) is defined inductively as follows. 

(1) V(a) = {(a, ok,)} where a 0 (v) = v for all v in N. 

(2) V(v,) = {(v„ a 0 )}. 

(3) Let r, and r 2 be rewbrs with values V(r,) and V{r 2 ). Then, 

(a) V{r x | r 2 ) = V(r,) (J V(r 2 ). 

( b ) V( (ri)(r 2 ) ) is the set of pairs (i,j 2 , «) such that (j,, a,) is in 

V(r,), (t, a 2 ) is in V(r 2 ), and s 2 is / with each v, in t replaced by 

oti(v,). That is to say, the assignment of strings to variables given 
by a, determines the string with which to replace each variable in 
t. The assignment a for the string s,j 2 is defined: a(v) = a 2 (v) 
unless a 2 (v) = v, in which case a(v) = a,(v). 

( c ) V ( ( r 0* ) = U v ( r 'i)- As before, r° = e and V(e) = {( e , a 0 )}. 

(d) V( (r,) ) = V(n). 

(e) V(r,.v,) is the set of pairs (r, a) such that for some a', (s, a') is 
in V(r,) and a(v,) = s and for all v # V/ , a(v) = a'(v). 

The pattern denoted by a rewbr r is the set of strings r in 2* such that 
ticm ^ m ^ r f ° r 801116 50016 exam P ,es should hel P clarify this defini- 

(1) 2* 2_v 2* v 2* denotes any string with at least one repeated charac¬ 
ter. To see this note that 2* matches any string of characters and that 
2.v will match any single character and assign v the value of that char¬ 
acter. The second variable v will then match a second occurrence of 
that character. 

(2) 2*.v v denotes any string of the form xx where x is any string of char¬ 
acters. 6 

(3) ((2*.v) v)* denotes any string of the form s lXl s 2 s 2 ■ ■ ■ s n s n where each 
Sj is a string in 2*. 

These examples illustrate some of the definitional power of rewbrs. 
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The pattern of example (1) can be denoted by an ordinary regular expres¬ 
sion, but only by a considerably longer expression. The pattern in example 
( 2 ) is not a regular or even context-free language so it cannot be expressed 
by a regular expression. Example (3) likewise cannot be expressed by a reg¬ 
ular expression or context-free grammar. Regular expressions with back 
referencing but with one variable name and no alternation have been studied 
by Angluin [5]. 


III. MATCHING FINITE SETS OF KEYWORDS 


This section considers the first of our three matching problems. We 
are given a pattern consisting of a finite set of keywords and an input string 
x. We wish to determine whether x has a substring that is a keyword in the 
pattern set. 


A. The Knuth-Morris-Pratt Algorithm 

A well-studied special case of this problem occurs when the pattern 
consists of exactly one nonempty keyword p. The straightforward way to 
look for a single keyword p in an input string x is by a program of the fol¬ 
lowing form. 

match(p, x) 

begin 

let p = *i* 2 • ■ • b m 
let x = a\ai • • • a„ 
i - 1 

while i < n-m + 1 do 

if bj = eij+j-i for 1 < 7 < m 
then return “yes” 
else i «- i + l 
return “no” 

end 

The worst-casfe performance of this algorithm requires mn-m 2 +m character 
comparisons. - (Consider the case where p = a m ~ l b and x = a" .) Knuth, 
Morris and Pratt discovered an elegant nonbacktracking algorithm that 
requires only 0(m + n) time [26]. The algorithm first constructs a table h 
from the keyword alone. Then the following procedure looks for the key¬ 
word in the input string. 
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match(p, x) 

begin 

let p - b\b 2 • ' b m 

let x = a x a 2 ■ '<!„ 

i -1 

j - 1 

while i n and j < m do 

begin 

while j > 0 and a x # bj do j - h\j ] 
i «- i + 1 
+1 

end 

if j > m then return “yes” 
else return “no” 

end 

This algorithm can be pictured as aligning the keyword above the input 
string and comparing the keyword with the portion of the input string under¬ 
neath. The key idea in the algorithm is that if we have matched the prefix 
b x b 2 • • • bj -1 of the keyword with the substring a,- J+l a i - ]+2 ■ • • «,■-1 of the 
text string and a, # bj , then we do not need to rescan a,-; + i ■ ' ■ «/-i s,nc ^ 
we know this portion of the input string is equal to the prefix of the keyword 
we have just matched. Instead, in each iteration of the inner while-loop we 
slide the pattern j-h[j] positions to the right and reset j to h\j] until ; 
becomes zero (in which case none of the pattern matches the current sub¬ 
string of the text string) or until a, = bj (in which case b x • • ■ bj- } bj 
matches • ■ • a,_ ja,). In the outer while-loop we resume the matching 

process by comparing a ,+1 with b J+ \. 

In order for this algorithm to work properly, the function h must have 
the property that h\j] is the largest k less than j such that b x b 2 ■ b k - X is a 
suffix of b x b 2 ■ ■ ■ bj- X (i.e., b x ■ ■ ■ b k -1 = V*+i ' ' ' b >~^ an ^ h > * 

If there is no such it, then h\j) = 0. The table h can be computed from the 
keyword by a program that is virtually identical to the matching program 
itself: 
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begin 


end 


i - 1 
7-0 

Mi]-o 

while / < m do 
begin 


while j > 0 and bj + bj do j - h\j ] 
i - i + 1 
7 -7 + 1 

if b t = bj then /i[i] - h\j] 
else /z[i] - j 


It is not hard to show that in both programs the assignment statement 
7 - h\j] in the inner loop is never executed more often than the statement 
i - i + 1 in the outer loop. Consequently, the program to compute h runs 
in 0{m) time and the program match runs in 0(n ) time. 

The existence of an 0(m+n) string-matching algorithm follows from 
Cook’s result that every two-way deterministic pushdown automaton 
(2DPDA) language can be recognized in linear time on a random access 
machine [9], The language {y#x \ y is a substring of *} can be recognized by 
a 2DPDA. Knuth, in fact, traced through the simulation implied in Cook’s 
result to derive the linear time pattern-matching algorithm. 

Knuth, Morris and Pratt give a fascinating historical account of the 
development of this algorithm along with many further refinements and 
suggestions for efficient implementation [26]. The algorithm has been used 
in a few text editors since the early 1970’s and some of the basic ideas were 
used by Gilbert in his work on comma-free codes in the early 1960’s [17]. 


B. Multiple Keyword Patterns 

Now consider the case where p = {y\, y 2 .y*}, a finite set of key¬ 

words rather than just one keyword. The straightforward way to look for 
multiple keywords in a string is to apply a single-keyword pattern-matching 
algorithm once for each keyword. In this section we shall outline a decidedly 
more efficient approach using ideas from finite automata theory and from 
the Knuth-Morris-Pratt algorithm. The language 

L = ' ' ' #y*##x I x = uy t v for some 1 < i < 1 } 

can be recognized by a 2DPDA, and consequently Cook’s theorem implies 
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that there is an 0(m + n) algorithm to do the matching, where m is the size 
°f P (i- e > the sum of the lengths of the keywords) and n is the length of x. 

An efficient way to do the pattern matching in 0(m+n) time is to first 
construct from the set of keywords a pattern-matching machine that will look 
for the keywords in parallel rather than one at a time. Various kinds of fin¬ 
ite automaton models can be used for this purpose. Here we use a pattern- 
matching automaton A that consists of the following components: 

(1) S, a finite set of states, 

(2) 2, a finite input alphabet, 

(3) g\ 5x2 - S |J {fail}, a forward transition function, 

(4) h : 5-{j 0 } - S, a backward transition function, 

(5) s 0 , an initial state, and 

(6) F, a set of accepting states. 

The pattern-matching automaton A processes an input string x in the 
following manner: 

match(/l, at) 

begin 

let x = aja 2 • ■ • a„ 
state - s 0 
i - 1 

while / < n do 
begin 

while g[state, a,] = fail do state - h[state] 
state - g[state, a t \ 
if state is in F then return “yes” 
end 

return “no” 

end 

The pattern-matching automaton starts off in the initial state scanning 
the first character of x. It then executes a sequence of moves. In one move 
on input symbol a, the pattern-matching automaton makes zero or more 
backward transitions until it reaches a state for which g[state, a,] * foil. 
The pattern-matching automaton then makes a forward transition to state 
g[state,aj]. If this state is an accepting state, the automaton halts and 
returns “yes”; otherwise, the automaton executes another move on the next 
input character a /+1 . 

The forward and backward transition functions of the pattern-matching 
automata we construct will have the following two properties: 

(1) g[j 0 , a] * fail, for all a in 2. 
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(2) If /i[j] j', then the depth of s' is less than the depth of j, where the 

depth of a state j is the length of the shortest sequence of forward tran¬ 
sitions from j 0 to s. 

. The , first Property guarantees that no backward transitions will occur in 
the initial state. The second property guarantees that the total number of 
backward transitions in processing an input string will be less than the total 
number of forward transitions. Since exactly one forward transition is made 
for each input character, less than 2 n transitions of both kinds will be made 
m Processing an input string of length n. Thus, the procedure matchM x) 
can be implemented to run in O(n) time. 

We must now show that we can construct the forward and backward 
transition functions of a pattern-matching automaton from the set of key¬ 
words in O(m) time. This can be done in the following manner. 

C U ^ J F,rst construct from p a trie in which the root is the initial state s Q . 
fcach node of the trie is a state s that corresponds to a prefix b,b 7 ■ ■ ■ b of 
some keyword y in p. We define g [s, V ,J . where ,■ corresponds' to 
me prefix b, ■ ■ ■ bjb j+l of y. Each state corresponding to a complete key¬ 
word becomes an accepting state. 

♦ d r° r St3te *° We define *1*0, a] = s 0 for all a for which g[s 0 , a] was 
not defined in step (1). 1 J 

* if) ^ = fail f ° r a11 5 and a for which *(■*> °] was not defined in 
steps (1) or (2). 

These three steps define the forward transition function Note that 
state s 0 has the property that g[s 0 , a] ± fail for any a in 2. 

Before we define the backward transition function, we define a failure 
function / as follows: 

(1) For all states j of depth one, /[j] = 

(2) Assume / has been defined for all states of depth d Let s d be a 

state of depth d such that g[s d , a] = . Then /[,'] is computed in the fol¬ 

lowing way: Let j - /[^]. Let t be the value 5 after executing the following 
statement: 6 


while g[j, a] = fail do -/[.s] 

Note that since g[j 0 , a] + fail, the while-loop will always terminate. Then 
/[ J ] = S[t, a). 

The failure function has the following key property: if states s and t 
represent prefixes u and v of some keywords, then /[i] = t if and only if v is 
the longest proper suffix of u that is also the prefix of some keyword in p. 

The failure function itself can be used as the backward transition func¬ 
tion. However, the failure function can cause some unnecessary backward 
transitions to occur. A more efficient backward transition function h that 
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avoids these redundant backward transitions can be constructed from / as fol¬ 
lows: 

(1) /i[j] = j 0 f° r all states s of depth one. 

(2) Assume h has been defined for all states of depth d. Let s be a 
state of depth d+ 1. If the set of characters on which there is a forward 
non-fail transition in state /[s] is a subset of the set of characters on which 
there is a forward non-fail transition in state s, then /i[s] = /i [/"[$]]; other¬ 
wise, /i[s] = /[$]. 

It is not hard to construct the function h from the set of keywords in 
O(m) time. See [1] for more details. 

Example. Let p = {aaa, abaaa, ababaaa). The forward transition function 
g , the failure function /, and the backward transition function h for this set 
of patterns are shown in Table I. 


/ h 


a _ b 


0 

1 

0 



1 

2 

4 

0 

0 

2 

3 

fail 

1 

1 

3 

fail 

fail 

2 

2 

4 

5 

fail 

0 

0 

5 

6 

8 

1 

0 

6 

7 

fail 

2 

1 

7 

fail 

fail 

3 

2 

8 

9 

fail 

4 

0 

9 

10 

fail 

5 

5 

10 

11 

fail 

6 

1 

1LJ 

fail 

fail 

7 

2 


Table I. 

To illustrate the distinction between using / and h for the backward 
transition function consider the behavior of the pattern-matching automaton 
after having read the first six characters of the input string ababaab. If the 
automaton started from initial state 0, then it will have reached state 10. 
Using / for the backward transition function, on the last input character the 
pattern-matching automaton would make backward transitions to states 6, 2, 
and 1 before making a forward transition to state 4. Using h for the back¬ 
ward transition function, on the last input character the pattern matching 
character would make just one backward transition to state 1 before making 
the forward transition to state 4. 
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Aho and Corasick used this pattern-matching algorithm in a biblio¬ 
graphic search system in which a user could specify documents by prescribing 
Boolean combinations of keywords and phrases. This approach yielded a sys¬ 
tem whose cost was 4 to 12 times faster than an identical system using a 
straightforward string-matching algorithm [1], This experience corroborates 
Knuth’s remarks that knowledge of automata theory can be useful in practi¬ 
cal problems [26], 

There are several interesting theoretical aspects to this algorithm. If 
the pattern consists of only one keyword of length m, then logbackward 
transitions are necessary and sufficient in any one move [26], Here 
c(> = (l + V5)/2, the golden ratio. This bound is achieved when the pattern 
is a Fibonacci string defined as follows: 

*i = b 

s 2 = a 

s k = Sk-\S k -2 

If the pattern is a set of keywords the sum of whose lengths is m, then 0(m) 
backward transitions may be necessary in a single move. Galil, on the other 
hand, has shown that pattern matching can be done in real-time on a random 
access machine by continuing to read ahead at the same time failure transi¬ 
tions are being made [14], 


C. The Boyer-Moore Algorithm 

Boyer and Moore give an interesting algorithm that looks for a match 
of a single keyword b\b 2 ■ b m in an input string a\a 2 • • • a n by ignoring 
those portions of the input string that cannot possibly contribute to a match 
[6], When the alphabet size is large, the algorithm determines whether a 
match occurs looking at only about n/m input string characters on the aver¬ 
age. Boyer and Moore’s approach is not readily modeled by conventional 
automata theory in that not all the input string is necessarily examined in 
determining a match. 

The basic idea of the algorithm is to look for a match by sliding the 
keyword across the input string from left to right and by comparing charac¬ 
ters in the keyword from right to left. Initially, we compare b m with a m . If 
a m occurs nowhere in the keyword, then we can slide the keyword m charac¬ 
ters to the right and try matching b m with a^. 

If a match occurs, we compare characters in the keyword with those in 
the text string from right to left until a match is verified or until a mismatch 
occurs. In case of a mismatch various strategies can be used to determine 
how far to shift the keyword to the right. The basic form of the Boyer- 
Moore algorithm is as follows: 
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match(p, x) 

begin 

let p = b x b 2 ■ ■ ■ b m 
let x = aia 2 • • a n 

i - m 

while i ^ n do 
begin 

j — m 

while j > 0 and a, = bj do 
begin 

i-i -1 

j~j -1 

end 

if j = 0 then return “yes” 
else i - i + max(</i[a,-], ^L/]) 

end 

return “no” 

end 


This algorithm uses two tables d x and d 2 to determine how far to slide 
the keyword to the right when a, # bj. Both tables can be computed in 
0(m) time by preprocessing the keyword. The first table is indexed by char¬ 
acters; d x [a\ is defined to be the largest j such that a = bj or m if a does not 
occur in the keyword. 

There are several ways of defining the second table d 2 which is indexed 
by positions in the keyword. Knuth [26] suggested the definition 

d 2 \j] = min {s+m-j \ s ==1 and (s s j or bj- s + bj ) 
and ((s s i or = b,) for 7 < i — m)} 

This table can be computed in O(m) time using the following algorithm. 
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begin 

for / - 1 to m do d 2 [i] - 2 *m-i 
j ~m 
k - m + l 
while j > 0 do 
begin 

m-k 

while k £ m and bj =£ b k do 
begin 

d 2 [k\ - min(d 2 [*], m-j) 

k -m 

end 

j -j ~ 1 

k ~k - 1 

end 

for i - 1 to k do d 2 [i] - min(d 2 [/fc], m+k-i) 

end 

The function / computed by this algorithm is the failure function of Section 
3.2 defined on the reversal of the keyword. 

Boyer and Moore’s original version of their algorithm had quadratic 
worst case behavior. However, for the version given above one can show the 
worse case behavior is linear in the length of the input string [13, 20, 26], 

A question of theoretical interest that has not yet been fully resolved is 
optimum string pattern matching. Rivest has shown that the minimum 
number of input string characters that must be examined to determine a 
match is n-m + 1 [33]. The minimum average number of characters that 
must be examined is not known. Nor is it known how to construct optimal 
average case algorithms. 

Recently, Commentz-Walter has described an approach by which the 
ideas in the Boyer-Moore algorithm can be combined with those of Section 
3.2 to look for patterns of finite sets of keywords in an input string in sub- 
linear time on the average [8]. The basic idea is to construct a trie as 
described in Section 3.2 for the set of keywords reversed. Matching then 
proceeds as in the Boyer-Moore algorithm with the trie playing the role of 
the single keyword. For the details of this approach see [8]. 
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IV. MATCHING REGULAR EXPRESSIONS 


Consider now the case where we are given a regular expression r and 
an input string x and we wish to determine whether jc contains a substring 
denoted by r. There are several approaches to answering this question. 


A. Nondeterministic Finite Automata 

One classical approach is to construct from the regular expression r a 
nondeterministic finite automaton (NDFA) M to do the pattern matching. 
The NDFA can take the form of an executable program [36] or a transition 
table that is interpreted by a simulator. The following recursive procedure 
can be used to construct M from r. 

(1) If r is a single character a construct the machine 

o—-o 

consisting of a transition on a from the initial state to the final state. 

(2) If r is of the form r x | r 2 , construct the machine 



Here we have created a new initial state and a new final state. There is a 
transition on the empty string from the new initial state to the initial states of 
A/, and M 2 , the machines for/, and r 2 . There is a transition on the empty 
string from the final state of M\ and M 2 to the newly created final state. 

(3) If r is of the form r x r 2 , construct the machine M 


Here we have merged the final state of Af,, the machine for r u with the ini¬ 
tial state of M 2 , the machine for r 2 . The initial state of Af, becomes the 
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initial state of M and the final state of M 2 becomes the final state of M. 

(4) If r is of the form R x *, construct the machine 



Here we have created a new initial state and a new final state. A transition 

of M e S y een n th T7 ^ initial St3te 3nd the initial state 

of between the final state of M j and the newly created final state 

between the new initial state and the new final state, and between the old 
final state and the old initial state. tne old 

useful pr^r Cti ° n ° f 3 NDFA * fr ° m 3 ^ « ' *- -era, 

(1) The number of states in M is at most twice the length of r. 

(2) No state has more the two transitions to other states. 

(3) There are no transitions out of the final state. 

. . Th£Se pro P erties enable us to simulate the behavior of M on * effi¬ 
ciently. Let m be the length of r and n the length of x . In the simulation 
we maintain a queue of states M can be in after having read a a 

Zrl ° f T Can COmpuK in °<"> «* set of Bate « 

can be in after having read a, since each state on the queue has at most two 
out-transitions. In .h,s way we can s i m „,a,e ,he behavior of u on , Tn 
0(mn) time and m apace. Chapter 9 of [2] contains the details of the simu- 


"• deterministic Finite Automata 

construcTfrom , appr ° ach “ regUlar “ pressi °" P 8 "'™ matching is to 

p ' l r ' from lhe expression a deterministic finite automaton (DFA) 

29). There are several ways of doing this. One way is to construct from the 

thftmi, v PreSSI ° n 3 IK,ndetermi " istic fMtt automaton as above, eliminate 
the transitions on empty strings, and then use the Rabin and Scott 132] suT 
^ construction to convert the resulting nondeterministic finite lutommon 

E state Te'T2lo n fW| a f U,0ma ' 0 V hal “ mOS ‘ one ™ >™nsition in 
eacn state. (See [2] or [22] for more details.) 
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A simpler and more direct way is to use the LR(0) parser construction 
technique to construct from the regular expression a deterministic finite auto¬ 
maton directly. Let t be the syntax tree [4] for the regular expression. Each 
state of the deterministic finite automaton corresponds to a set of leaves of 
the syntax tree that can be active at a given point in time in much the same 
way as each state of a DFA constructed from a NDFA using the subset con¬ 
struction corresponds to a set of nondeterministic states that can be active at 
a given point in time. For example, the syntax tree for the regular expres¬ 
sion 1*(abc | aba ) is 



Initially, leaves 1, 2, and 5 are active so the initial state would be the set 
{1,2,5}. From the initial state there would be a transition on input character 
a to state {1,3,6}, the set of leaves that would be active after the DFA had 
read an a. 

A DFA can be simulated efficiently by a program since it makes 
exactly one transition for each input character. Thus, once a DFA has been 
constructed from the regular expression, it can do the pattern matching in 
O(n) steps. Note that neither the size of the input alphabet nor the length of 
the regular expression need affect the recognition speed. The main problem 
with using a DFA for pattern matching, however, is that the DFA may be 
time consuming to construct from the regular expression and that the transi¬ 
tion function of a DFA may require a lot of storage. If the transition func¬ 
tion is stored as a two-dimensional array, then the storage requirement will 
be the product of the alphabet size times the number of states. In many 
practical applications this can be excessive. Consequently, several interesting 
sparse matrix techniques have been developed to reduce the space require¬ 
ments for the transition function. See [4, 10, 35] for descriptions of some of 
these techniques. 

However, even the sparse matrix techniques cannot cope with an 
exponential number of states. For example, consider the regular expression 
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consisting of S*a followed by m-1 2’s. This common regular expression 
denotes all strings in which the mth character from the right end is an a. 
Unfortunately the smallest deterministic finite automaton that recognizes this 
pattern has 2 m states. 

C. Hybrid Deterministic-Nondeterministic Finite Automata 

Given a regular expression of length m and an input string of length n , 
we can use a NDFA to do pattern matching in time 0{mn) and space O(m). 
With a DFA we can do pattern matching in time 0(2 m + n) and space 
0(2 m ). Lower bounds and optimal algorithms for regular expression pattern 
matching are not known. It would be interesting to know more precise 
bounds on the difficulty of regular expression pattern matching. Fischer and 
Paterson [12] show that a single string pattern with don’t care symbols (2’s) 
can be matched in O(nlogmloglogwi) time using the Schonhage-Strassen algo¬ 
rithm [2] for integer multiplication as a subroutine. It would also be interest¬ 
ing to know whether there exists a Boyer-Moore type algorithm for regular 
expression pattern matching. 

The linear space requirements of the NDFA and the linear recognition 
speed of the DFA do suggest a hybrid deterministic-nondeterministic 
approach. In such a scheme we construct a NDFA from the regular expres¬ 
sion. We then make the most frequently visited states of the NDFA into 
deterministic states by using the subset construction. In this way we can con¬ 
struct a finite automaton in which the most frequently visited states have at 
most one transition on any input character. These frequently visited states 
can be implemented as indexable arrays so that transitions from these states 
can be executed in constant time. Myers has discovered that in practice with 
this approach one can closely approximate the time efficiency of DFA’s and 
the space efficiency of NDFA’s in regular expression pattern matching [31]. 
The theoretical optimal average case behavior of these hybrid machines is 
still unknown. 


V. MATCHING REGULAR EXPRESSIONS 
WITH BACK REFERENCING 


Variables make matching regular expressions with back referencing 
much more difficult that matching ordinary regular expressions. The most 
straightforward approach to matching a rewbr pattern r is to use backtrack¬ 
ing to keep track of the possible substrings of the input string x that can be 
assigned to the variables in r. There are 0(n 2 ) possible substrings that can 
be assigned to any one variable in r, where n is the length of x. If there are 
k variables in r, then there are ^(n 2 *) possible assignments in all. Once an 
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assignment of substrings to variables is fixed, the problem reduces to ordinaty 
regular expression matching. Thus, rewbr matching can be done in at worst 
CKn 2 *) time. 

One might wish for a more efficient process. Unfortunately, the prob¬ 
lem of determining whether a rewbr r matches an input string x is NP- 
complete. The following example illustrates a reduction from the vertex 
cover problem. 

Let E\, £ 2 .£ m be subsets of cardinality two of some finite set 2. 

The vertex cover problem is to determine given a positive integer k whether 
there exists a subset 2' of 2 of cardinality at most k such that 2' contains at 
least one element in each £,, We can think of the £,’s as being edges of a 
graph and 2' as being a set of vertices such that each edge contains at least 
one node in 2'. The vertex cover problem is a well-known NP-complete 
problem [2, 16, 23]. 

We can transform this problem into a matching problem for rewbrs as 

follows. Let 5, s u s 2 .s m be strings consisting of the elements in 2, 

Ex E 2 . £ m , respectively. Let # be a new symbol not in 2. For 

1 < i < it, let 

x, = 2* 2.v, 2*# 

For 1 < i < m, let 

y, = 2* 2.w, 2*# 

For 1 < / < m, let 

Zj = W,*V 1 *V 2 * • • • v t *w,*# 

Now, let r be the rewbr x x 


X k y\ ■ ■ ■ JU*1 • • • z m- 

We shall now construct an input string. Let f 0 te the strin § s * 
repeated k+m times. For 1 < i ^ m, let t, be s,#. Let u be the input 
string tgfi • • • t m . 

Consider what happens when we use r to match u. r matches u if an 
only if there exists an assignment of elements of 2 to the variables 
v, v* such that for each i there exists a j such that vj is in £,, that is, it 
and only if the subset {v,.vj of 2 is a vertex cover for £,. E m . 

This NP-completeness result strongly suggests that the time complexity 
for rewbr matching is inherently exponential in the worst case. Practical 
experience, on the other hand, reasonable performance for the rewbr pat¬ 
terns seen by the program grep. For example, on a DEC PDP-li/7 0 using 
the UNIX program grep to search Webster’s second (with 2486813 characters 
in 234936 words) for those words in which no letter is repeated (of which 
dermatoglyphics is the longest) requires 9.7 min whereas to search for those 
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words in which the letters are monotonically increasing (of which egilops is 
the longest) requires 3.8 min. The UNIX program egrep which simulates a 
DFA will find the latter pattern in 1.1 min. 


VI. CONCLUSIONS 

In this paper we have examined three classes of patterns that are basic 
to common text-processing applications. We have seen that there are several 
algorithms that can be used to search for these patterns. Knowledge of finite 
automata theory has proven useful in the design of several of these algo¬ 
rithms. Since no one algorithm seems to be ideal for all situations, it is 
necessary to examine the time and space requirements of the application at 
hand to determine the algorithm best suited for that situation. 

Although we have limited our discussion to string-matching algorithms, 
we should point out that in many text-processing applications it is necessary 
to be able to look for patterns that combine both textual and numeric data. 
Some higher-level text-processing languages such as AWK [3] and POPLAR 
[30] have been developed for such applications recently. In such languages 
the algorithms described in this paper can be used to speed up the string 
matching process. 
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The LR syntax analysis method is a useful and versatile technique for parsing 
deterministic context-free languages in compiling applications. This paper 
provides an informal exposition of LR parsing techniques emphasizing the 
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apply to ambiguous grammars. 
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1. INTRODUCTION 

A complete specification of a programming 
language must perform at least two func¬ 
tions. First, it must specify the syntax of the 
language; that is, which strings of symbols 
are to be deemed well-formed programs. 
Second, it must specify the semantics of the 
language; that is, what meaning or intent 
should be attributed to each syntactically 
correct program. 

A compiler for a programming language 
must verify that its input obeys the syntactic 
conventions of the language specification. It 
must also translate its input into an object 
language program in a manner that is con¬ 
sistent with the semantic specification of the 
language. In addition, if the input contains 
syntactic errors, the compiler should an¬ 
nounce their presence and try to pinpoint 
their location. To help perform these func¬ 
tions every compiler has a device within it 
called a parser. 

A context-free grammar can be used to 
help specify the syntax of a programming 
language. In addition, if the grammar is de¬ 
signed carefully, much of the semantics of 
the language can be related to the rules of 
the grammar. 

There are many different types of parsers 
for context-free grammars. In this paper we 


shall restrict ourselves to a class of parsers 
known as LR parsers. These parsers are 
efficient and well suited for use in compilers 
for programming languages. Perhaps more 
important is the fact that we can automati¬ 
cally generate LR parsers for a large and use¬ 
ful class of context-free grammars. The pur¬ 
pose of this article is to show how LR parsers 
can be generated from certain context-free 
grammars, even some ambiguous ones. An 
important feature of the parser generation 
algorithm is the automatic detection of 
ambiguities and difficult-to-parse constructs 
in the language specification. 

We begin this paper by showing how a 
context-free grammar defines a language. 
We then discuss LR parsing and outline the 
parser generation algorithm. We conclude 
by showing how the performance of LR 
parsers can be improved by a few simple 
transformations, and how error recovery and 
“semantic actions” can be incorporated into 
the LR parsing framework. 

For the purposes of this paper, a sentence 
is a string of terminal symbols. Sentences are 
written surrounded by a pair of single quotes. 
For example, ‘a’, ‘ab’, and V are sentences. 
The empty sentence is written Two sen¬ 
tences written contiguously are to be con¬ 
catenated, thus ‘a’ ‘b’ is synonymous with 
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‘ab’. In this paper the term language merely 
means a set of sentences. 


CONTENTS 



2. GRAMMARS 

A grammar is used to define a language and 
to impose a structure on each sentence in 
the language. We shall be exclusively con¬ 
cerned with context-free grammars, sometimes 
called BNF (for Backus-Naur form) specifi¬ 
cations. 

In a context-free grammar, we specify 
two disjoint sets of symbols to help define a 
language. One is a set of nonterminal symbols. 
We shall represent a nonterminal symbol by 
a string of one or more capital roman letters. 
For example, LIST represents a nonterminal 
as does the letter A. In the grammar, one 
nonterminal is distinguished as a start (or 
sentence) symbol. 

The second set of symbols used in a con¬ 
text-free grammar is the set of terminal 
symbols. The sentences of the language gen¬ 
erated by a grammar will contain only 
terminal symbols. We shall refer to a termi¬ 
nal or nonterminal symbol as a grammar 
symbol. 

A context-free grammar itself consists of a 
finite set of rules called productions. A 
production has the form 

left-side -* right-side, 

where left-side is a single nonterminal symbol 
(sometimes called a syntactic category) and 
right-side is a string of zero or more grammar 
symbols. The arrow is simply a special 
symbol that separates the left and right 
sides. For example, 

LIST — LIST V ELEMENT 


Copyright © 1974, Association for Computing 
Machinery, Inc. General permission to republish, 
but not for profit, all or part of this material is 
granted, provided that ACM’s copyright notice is 
given and that reference is made to this publica¬ 
tion, to its date of issue, and to the fact that re¬ 
printing privileges were granted by permission of 
the Association for Computing Machinery. 


is a production in which LIST and ELE¬ 
MENT are nonterminal symbols, and the 
quoted comma represents a terminal sym¬ 
bol. 

A grammar is a rewriting system. If aAy 
is a string of grammar symbols and A —> 0 
is a production, then we write 

aAy => a(3y 

and say that aAy directly derives a0y. A 
sequence of strings 
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oto, ai, • ■ • , a„ 

such that a,_i => a, for 1 ^ i ^ n is said to 
be a derivation of a, from a 0 . We sometimes 
also say a. is derivable from a 0 . 

The start symbol of a grammar is called a 
sentential form. A string derivable from the 
start symbol is also a sentential form of the 
grammar. A sentential form containing only 
terminal symbols is said to be a sentence 
generated by the grammar. The language 
generated by a grammar G, often denoted 
by L(G), is the set of sentences generated by 
G. 

Example 2.1: The following grammar, 
hereafter called Gi, has LIST as its start 
symbol: 

LIST LIST 7 ELEMENT 
LIST -> ELEMENT 
ELEMENT ‘a’ 

ELEMENT — V 
The sequence: 

LIST => LIST y ELEMENT 
=» LIST ‘,a’ 

=> LIST 7 ELEMENT \a’ 

=> LIST ‘,b,a’ 

=> ELEMENT ‘,6, a’ 

=> ‘a,b,a’ 

is a derivation of the sentence ‘a,b,a’. L(Gi) 
consists of nonempty strings of a’s and b’s, 
separated by commas. 

Note that in the derivation in Example 
2.1, the rightmost nonterminal in each sen¬ 
tential form is rewritten to obtain the fol¬ 
lowing sentential form. Such a derivation is 
said to be a rightmost derivation and each sen¬ 
tential form in such a derivation is called a 
right sentential form. For example, 

LIST ‘,b,a’ 

is a right sentential form of f\. 

If aAw is a right sentential form in which 
w is a string of terminal symbols, and aAw=> 
a0w, then 0 is said to be a handle of a0w* 
For example, ‘b’ is the handle of the right 
sentential form 

LIST ‘,6,o’ 

in Example 2.1. 

* Some authors use a more restricting definition of 
handle. 


A prefix of af3 in the right sentential form 
af3w is said to be a viable prefix of the gram¬ 
mar. For example, 

LIST 7 

is a viable prefix of Gi, since it is a prefix of 
the right sentential form, 

LIST 7 ELEMENT 
(Both a and w are null here.) 

Restating this definition, a viable prefix of 
a grammar is any prefix of a right sentential 
form that does not extend past the right end 
of a handle in that right sentential form. 
Thus we know that there is always some 
string of grammar symbols that can be ap¬ 
pended to the end of a viable prefix to ob¬ 
tain a right sentential form. Viable prefixes 
are important in the construction of com¬ 
pilers with good error-detecting capabilities; 
as long as the portion of the input we have 
seen can be derived from a viable prefix, 
we can be sure that there are no errors that 
can be detected having scanned only that 
part of the input. 


3. DERIVATION TREES 

Frequently, our interest in a grammar is 
not only in the language it generates, but 
also in the structure it imposes on the sen¬ 
tences of the language. This is the case be¬ 
cause grammatical analysis is closely con¬ 
nected with other processes, such as compila¬ 
tion and translation, and the translations or 
actions of the other processes are frequently 
defined in terms of the productions of the 
grammar. With this in mind, we turn our 
attention to the representation of a deriva¬ 
tion by its derivation tree. 

For each derivation in a grammar we can 
construct a corresponding derivation tree. 
Let us consider the derivation in Example 
2.1. To model the first step of the derivation, 
in which LIST is rewritten as 

LIST 7 ELEMENT 

using production 1, we first create a root 
labeled by the start symbol LIST, and then 
create three direct descendants of the root, 
labeled LIST, and ELEMENT: 
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(We follow historical usage and draw our 
“root” node at the top.) In the second step 
of the derivation, ELEMENT is rewritten 
as ‘a’. To model this step, we create a direct 
descendant labeled ‘a’ for the node labeled 
ELEMENT: 


( list"") (v) (element ) 

( 2 ) 


Continuing in this fashion, we obtain the 
following tree: 



Note that if a node of the derivation tree is 
labeled with a nonterminal symbol A and its 
direct descendants are labeled Xi, Xt, • • • , 
X n , then the production: 

A -* Xi X t • • • X n 

must be in the grammar. 

If 01 , 02 , ■ ■ ■, dm are the labels of all the 
leaves of a derivation tree, in the natural 
left-to-right order, then the string 

0102 • •• a m 

is called the frontier of the tree. For example, 
‘a,6,o’ is the frontier of the previous tree. 
Clearly, for every sentence in a language 


there is at least one derivation tree with 
that sentence as its frontier. A grammar that 
admits two or more distinct derivation trees 
with the same frontier is said to be ambigu¬ 
ous. 

Example 3.1: The grammar G t with pro¬ 
ductions 


LIST LIST 7 LIST 
LIST — ‘o’ 

LIST — ‘6’ 

is ambiguous because the following two 
derivation trees have the same frontier. 


LIST ) 

( L| S T ) (v) 


__ ST) 

WC LIST 5c5c LIST ) 

<sT gT 



LIST ) 

LIST ) (V) LIST ) 
LIST 


c l ' st ) O 

(b) 


In certain situations ambiguous grammars 
can be used to represent programming 
languages more economically than equiva¬ 
lent unambiguous grammars. However, if an 
ambiguous grammar is used, then some other 
rules should be specified along with the 
grammar to determine which of several 
derivation trees is to be associated with a 
given input. We shall have more to say 
about ambiguous grammars in Section 7. 


4. PARSERS 

We can consider a parser for a grammar to be 
a device which, when presented with an 
input string, attempts to construct a deriva- 
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tion tree whose frontier matches the input. 
If the parser can construct such a derivation 
tree, then it will have verified that the input 
string is a sentence of the language generated 
by the grammar. If the input is syntactically 
incorrect, then the tree construction process 
will not succeed and the positions at which 
the process falters can be used to indicate 
possible error locations. 

A parser can operate in many different 
ways. In this paper we shall restrict ourselves 
to parsers that examine the input string 
from left to right, one symbol at a time. 
These parsers will attempt to construct the 
derivation tree “bottom-up”; i.e., from the 
leaves to the root. For historical reasons, 
these parsers are called LR parsers. The “L” 
stands for “left-to-right scan of the input”; 
the “R” stands for “rightmost derivation.” 
We shall see that an LR parser operates by 
reconstructing the reverse of a rightmost 
derivation for the input. In this section we 
shall describe in an informal way how a cer¬ 
tain class of LR parsers, called LR(1) 
parsr-s, operate. 

An LR parser deals with a sequence of 
partially built trees during its tree construc¬ 
tion process. We shall loosely call this se¬ 
quence of trees a forest. In our framework the 
forest is built from left to right as the input 
is read. At a particular stage in the construc¬ 
tion process, we have read a certain amount 
of the input, and we have a partially con¬ 
structed derivation tree. For example, sup¬ 
pose that we are parsing the input string 
‘a,b’ according to the grammar G\. After 
reading the first ‘a’ we construct the tree: 



Then we construct: 



using the production, 

ELEMENT -► ‘a’ 

To reflect this parsing action, we say that ‘a’ 
is reduced to ELEMENT. Next we use the 
production 

LIST -*• ELEMENT 
to obtain the tree: 


C U ST ) 

I 



$ 


Here, ELEMENT is reduced to LIST. We 
then read the next input symbol and 
add it to the forest as a one node tree: 


( UST ) 



We now have two trees. These trees will 
eventually become sub-trees in the final 
derivation tree. We then read the next input 
symbol ‘b’ and create a single node tree for 
it as well: 



(s) © © 

Using the production, 

ELEMENT — ‘b’ 
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we reduce V to ELEMENT to obtain: 


( LIST ) 

L 



© O ® 


Finally, using the production 

LIST — LIST ',’ ELEMENT 

we combine these three trees into the final 
tree: 


In a reduce action, a production, such as 
A — Xi X* • • • X n 

is specified; each Xi represents a terminal or 
nonterminal symbol. A reduction by this 
production causes the following operations: 

(1) A new node labeled A is created. 

(2) The rightmost n roots in the forest 
(which will have already been labeled 
Xi, Xt, • • • , Xi) are made direct 
descendants of the new node, which 
then becomes the rightmost root of the 
forest. 

If the reduction is by a production of the 
form 

A 


C 


QjsT) 

(element) 

<r 


J 


6 


(element) 

<r 


At this point the parser detects that we have 
read all of the input and announces that the 
parsing is complete. The rightmost deriva¬ 
tion of ‘a, b’ in Gi is 

LIST => LIST 7 ELEMENT 
=* LIST ‘ ,b ’ 

* => ELEMENT ‘,b’ 

=> ‘o,6’ 

In parsing ‘a,6’ in the above manner, all we 
have done is reconstruct this rightmost 
derivation in reverse. The sequence of pro¬ 
ductions encountered in going through a 
rightmost derivation in reverse is called a 
right parse. 

There are four types of parsing actions 
that an LR parser can make; shift, reduce, 
accept (announce completion of parsing), or 
announce error. 

In a shift action, the next input symbol is 
removed from the input. A new node labeled 
by this symbol is added to the forest at the 
right as a new tree by itself. 


(i.e., where the right side is the empty 
string), then the parser merely creates a root 
labeled A with no descendants. 

A parser operates by repeatedly making 
parsing actions until either an accept or error 
action occurs. 

The reader should verify that the follow¬ 
ing sequence of parsing actions builds the 
parse tree for ‘a,6’ in G\: 

(1) Shift ‘o’ 

(2) Reduce by: ELEMENT — ‘o’ 

(3) Reduce by: LIST -► ELEMENT 

(4) Shift 7 

(5) Shift ‘6’ 

(6) Reduce by: ELEMENT —> ‘6’ 

(7) Reduce by: LIST — LIST ',’ 

ELEMENT 

(8) Accept 

We now consider the question of how an LR 
parser decides what parsing actions to make. 
Clearly a parsing action can depend on what 
actions have already been made and on what 
the next input symbols are. An LR parser 
that looks at only the next input symbol to 
decide which parsing action to make is 
called an LR(1) parser. If it looks at the 
next k input symbols, k ^ 0, it is called an 
LR(ft) parser. To help to make its parsing 
decisions, an LR parser attaches to the root 
of each tree in the forest a number called a 
state. The number on the root of the right¬ 
most tree is called the current state. In addi¬ 
tion, there is an initial state to the left of the 
forest, which helps determine the very first 
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parsing action. We shall write the states in 
parentheses above the associated roots. For 
example, 


(1) 



shift move or a reduce move, the parser 
must determine what state to attach to the 
root of the tree that has just been added to 
the forest. In a shift move, this state is de¬ 
termined by the current state and the input 
symbol that was just shifted into the forest. 

For example, if we have just shifted 
into the forest 


( 1 ) 

( UST ) 


represents a forest with states. State 5 is the 
current state, and state 0 is the initial state. 
The current state and the next input symbol 
determine the parsing action of an LR(1) 
parser. 

The following table shows the states of an 
LR(1) parser for G\, and the associated pars¬ 
ing actions. In this table there is a column 
labeled ‘S’ with special significance. The T 
stands for the right endmarker, which is 
assumed to be appended to the end of all 
input strings. Another way of looking at this 
is to think of ‘S’ as representing the condi¬ 
tion “Where we have read and shifted all of 
the “real” characters in the input string. 


0 

1 

2 

Current 3 
State 4 

5 

6 


Fig. 1 . Parsing Action Table for <7, 


Next Input Symbol 


shift 

error 

error 

error 

error 

shift 

error 


shift 

error 

error 

error 


error 
shift 
Red. 2 
Red. 3 
Red. 4 
error 
Red. 1 


error 
accept 
Red. 2 
Red. 3 
Red. 4 

Red. 1 


The reduce actions are represented as 
Red. n” in the above table; the integer n 
refers to the productions as follows: 

(1) LIST -♦ LIST 7 ELEMENT 

(2) LIST — ELEMENT 

(3) ELEMENT -► ‘ a’ 

(4) ELEMENT -* ‘6’ 

We shall refer to the entry for row s and 
column c as pa(s,c). After making either a 


( 0 ) 




then state 1 and V determine the state to be 
attached to the new rightmost root 
In a reduce move, suppose we reduce by 
production 


A. —* XiXf ■ • X n 

When we make nodes X u • • • , X n direct 
descendants of the root A, we remove the 
states that were attached to X u ■ ■ ■ , X„. 
The state that is to be attached to node A 
is determined by the state that is now the 
rightmost state in the forest, and the non¬ 
terminal A. For example, if we have just 
reduced by the production 

LIST -*• LIST 7 ELEMENT 
and created the forest 



101 &) 6 G) 

then state 0 and the nonterminal LIST de¬ 
termine the state to be attached to the root 
LIST. Note that the states previously at¬ 
tached to the direct descendants of the new 
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root have disappeared, and play no role in 
the calculation of the new state. 

The following table determines these new 
states for G\. For reasons that will become 
apparent later, we shall call this table the 
goto table for Gi. 


RKHTHOST 

STATE 


OF NEW ROOT 



Fio. 2. Goto Table for G, 


We shall refer to the entry in the row for 
state s and column c as goto(s, c). It turns 
out that the entries in the goto table which 
are blank will never be consulted [Aho and 
Ullman (1972b)]. 

An LR parser for a grammar is completely 
specified when we have given the parsing 
action table and the goto table. We can 
picture an LR(1) parser as shown in Fig. 3. 



Fio. 3. Pictorial Representation of an LR(1) 

Parser. 

The LR(1) parsing algorithm can be sum¬ 
marized as follows: 

Initialize: Place the initial state into an 
otherwise empty forest; the initial state is 
the current state at the beginning of the 
parse. 

Parsing Action: Examine the parsing ac¬ 
tion table, and determine the entry cor¬ 


responding to the current state and the 
current input symbol. On the basis of this 
entry (Shift, Reduce, Error, or Accept) do 
one of the following four actions: 

Shift: Add a new node, labeled with the 
current input symbol, to the forest. Associ¬ 
ate the state . 

goto(current state, input) 

to this node and make this state the new cur¬ 
rent state. Advance the input cursor to 
read the next character. Repeat the step 
labeled Parsing Action. 

Reduce: If the indicated production is 

A -> Xx X t ■ ■ ■ X n 

add a new node labeled A to the forest, and 
make the rightmost n roots, n ^ 0, direct 
descendants of this new node. Remove the 
states associated with these roots. If s is the 
state which is now rightmost in the forest 
(on the root immediately to the left of the 
new node), then associate the state 

goto(s,A) 

with the new node. Make this state the new 
current state. (Notice that the input charac¬ 
ter is not changed.) Repeat the step labeled 
Parsing Action. 

Accept: Halt. A complete derivation tree 
has been constructed. 

Error: Some error has occurred in the 
input string. Announce error, and then try 
to resume parsing by recovering from the 
error. (This topic is discussed in Section 9.) 

To see how an LR parser works, let us 
again parse the input string ‘a,b’ using the 
parsing action function pa (Figure 1) and 
the goto function (Figure 2). 

Initialization: We place state 0 into the 
forest; 0 becomes the current state. 

Parsing Action 1: pa(0, ‘a’) = shift. We 
create a new root labeled ‘a’ and attach state 
3 to it (because goto(0, ‘a’) = 3). We have: 


(3) 

<°> © 
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Parsing Action 2: pa(3, 7) = reduce 3. 
We reduce by production (3) 

ELEMENT — ‘a’ 

We examine the state immediately to the 
left; this is state 0. Since goto(0, ELE¬ 
MENT) = 2, we label the new root with 2. 
We now have: 


( 2 ) 



Parsing Action 3: pa(2, 7) = reduce 2. 
We reduce by production (2) 

LIST — ELEMENT 

goto(0, LIST) = 1, so the new state is 1. 

Parsing Action 4: p a (l, 7) = shift. We 
shift and attach state 5. 

Parsing Action 5: pa(5, V) = shift. We 
shift and attach state 4. We now have 


(1) 



Parsing Action 6: pa(4, “$’) = reduce 4. 
We reduce by production (4) 

ELEMENT V 

goto(5, ELEMENT) = 6, so the new state 
is 6. We now have 


( 1 ) 



Parsing Action 7: pa(6, '$’) = reduce 1. 
We reduce by production (1) 

LIST — LIST 7 ELEMENT 

The state to the left of the newly created 
tree is state 0, so the new state is goto(0, 
LIST) = 1. 

Parsing Action 8: pa(l, '$’) = accept. We 
halt and terminate the parse. 

The reader is urged to follow this pro¬ 
cedure with another string, such as ‘a,b,a’ to 
verify his understanding of this process. It is 
also suggested that he try a string which is 
not in L(Gi), such as ‘a,ba’ or ‘a„b’, to see 
how the error detection mechanism works. 
Note that the grammar symbols on the roots 
of the forest, concatenated from left to r igh t., 
always form a viable prefix. 

Properly constructed LR(1) parsers can 
parse a large class of useful languages called 
the deterministic context-free languages. These 
parsers have a number of notable properties: 

(1) They report error as soon as possible 
(scanning the input from left to right). 

(2) They parse a string in a time which is 
proportional to the length of the 
string. 

(3) They require no rescanning of previ¬ 
ously scanned input (backtracking). 

(4) The parsers can be generated mechan¬ 
ically for a wide class of grammars, 
including all grammars which can be 
parsed by recursive descent with no 
backtracking [Knuth (1971)] and 
those grammars parsable by operator 
precedence techniques [Floyd (1963)]. 

The reader may have noticed that the 
states can be stored on a pushdown stack, 
since only the rightmost state is ever used 
at any stage in the parsing process. In a 
shift move, we stack the new state. In a 
reduce move, we replace a string of states on 
top of the stack by the new state. 

For example, in parsing the input string 
‘ a,b ’ the stack would appear as follows at 
each of the actions referred to above. (The 
top of the stack is on the right.) 

Action stack input 

Initial 0 ‘a 6$’ 

1 0 3 ‘,65’ 

2 0 2 ‘,6J- 

3 0 1 ‘,65’ 

4 0 15 ‘65’ 







108 


A. V. Aho and S. C. Johnson 


Action Slack Input 

5 0 1 5 4 ‘S’ 

6 0 1 5 6 ‘S’ 

7 0 1 ‘S’ 

8 0 1 ‘S’ 

Thus, the parser control is independent of 
the trees, and depends only on a stack of 
states. In practice, we may not need to con¬ 
struct the derivation tree explicitly, if the 
translation being performed is sufficiently 
simple. For example, in Section 10, we men¬ 
tion a class of useful translations that can 
be performed by an LR parser without re¬ 
quiring the forest to be maintained. 

If we wish to build the derivation tree, we 
can easily do so by stacking, along with each 
state, the root of the tree associated with that 
state. 


5. REPRESENTING THE PARSING ACTION AND 
GOTO TABLES 

Storing the full action and goto tables 
straightforwardly as matrices is extremely 
wasteful of space for large parsers. For ex¬ 
ample, the goto table is typically nearly all 
blank. In this section we discuss some simple 
ways of compacting these tables which lead 
to substantial savings of space; in effect, we 
are merely representing a sparse matrix more 
compactly, using a particular encoding. 

Let us begin with the shift actions. If x is 
a terminal symbol and s is a state, the parsing 
action on x in state s is shift if and only if 
goto(s, x) is nonblank. We will encode the 
goto into the shift action, using the notation 

shift 17 

as a shorthand for “shift and attach state 17 
to the new node.” By encoding the gotos on 
terminal symbols as part of the action table, 
we need only consider the gotos on non¬ 
terminal symbols. We will encode them by 
columns; i.e., by nonterminal symbol name. 
If, on a nonterminal symbol A, there are 
nonblank entries in the goto table corre¬ 
sponding to states #i, s 2 , • • • , s„, and we have 
s/ = goto(s,, A), for i = 1, • • • , n 
then we shall encode the column for A in a 
pseudo-programming language: 


A : if (state = si) goto = s\ 
if (state = s n ) goto = s,' 

The goto table of Gi would be represented in 
this format as: 

LIST: if (state = 0) goto = 1 
ELEMENT: if (state = 0) goto = 2 
if (state = 5) goto = 6 

It turns out that [Aho and Ullman (1972b)] 
whenever we do a goto on A, the state will 
always be one of si, • • • , s», even if the input 
string is in error. Thus, one of these branches 
will always be taken. We shall return to this 
point later in this section. 

We shall encode parsing actions in the 
same spirit, but by rows of the table. The 
parsing actions for a state s will also be 
represented by a sequence of pseudo-pro¬ 
gramming language statements. If the input 
symbols ai, • • •, a„ have the associated 
actions actioni, • • • , action,, then we will 
write: 

s: if (input = ffli) actioni 

if (input = a„) action. 

As we mentioned earlier, we shall attach 
goto(s,a,) onto the action if action* is shift. 
Similarly, if we have a reduction by the 
production A —> a, we will usually write 

reduce by A —► a 

as the action. 

For example, the parsing actions for state 
1 in the parser for G\ are represented by: 

1: if (input = ‘a’) error 
if (input = ‘b’) error 
if (input = V) shift 5 
if (input = ‘S’) accept 

At first glance this is no saving over the 
table, since the parsing action table is 
usually nearly full. We may make a large 
saving, however, by introducing the notion 
of a default action in the statements. A 
default action is simply a parsing action 
which is done irrespective of the input char¬ 
acter; there may be at most one of these in 
each state, and it will be written last. Thus, 
in state 1 we have two error actions, a shift 
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action, and an accept action; we shall make 
the error action the default. We will write: 

1: if (input = V) shift 5 
if (input = $ ) accept 
error 

There is an additional saving which is 
possible. Suppose a state has both error and 
reduce entries. Then we may replace all 
error entries in that state by one of the re¬ 
duce entries. The resulting parser may make 
a sequence of reductions where the original 
parser announced error but the new parser 
will announce error before shifting the next 
input symbol. Thus both parsers announce 
error at the same position in the input, but 
the new parser may take slightly longer be¬ 
fore doing so. 

There is a benefit to be had from this modi¬ 
fication; the new parsing action table will re¬ 
quire less space than the original. For 
example, state 2 of the parsing action table 
for G i would originally be represented by: 

2: if (input = ‘a’) error 
if (input = ‘ b’) error 
if (input = V) reduce 2 
if (input = T) reduce 2 

Applying this transformation, state 2 would 
be simply represented as: 

2: reduce 2 

Thus in a state with reduce actions, we 
will always have the shift and accept actions 
precede the reduce actions. One of the reduce 
actions will become a default action, and we 
will ignore the error entries. In a state with¬ 
out reduce actions, the default action will be 
error. We shall discuss other means of cut¬ 
ting down on the size of a parser in Section 8. 


6. CONSTRUCTION OF A PARSER FROM A 
GRAMMAR 

How do we construct the parsing action and 
goto tables of an LR(1) parser for a given 
grammar? In this section we outline a 
method that works for a large class of 
grammars called the lookahead LR(1) 
(LALR(l)) grammars. 

The behavior of an LR parser, as described 
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in the last section, is dictated by the current 
state. This state reflects the progress of the 
parse, i.e., it summarizes information about 
the input string read to this point so that 
parsing decisions can be made. 

Another way to view a state is to consider 
the state as a representative of an equiva¬ 
lence class of viable prefixes. At every stage 
of the parsing process-, the string formed by 
concatenating the grammar symbols on the 
roots of the existing subtrees must be a vi¬ 
able prefix; the current state is the repre¬ 
sentative of the class containing that viable 
prefix. 

6.1 Sets of Items 

In the same way that we needed to discuss 
partially built trees when talking about pars¬ 
ing, we will need to talk about “partially 
recognized productions” when we talk about 
building parsers. We introduce the notion of 
item* to deal with this concept. An item is 
simply a production with a dot (.) placed 
somewhere in the right-hand side (possibly 
at either end). For example, 

[LIST — LIST . ELEMENT] 
[ELEMENT — . ‘a’] 

are both items of Gi. 

We enclose items in square brackets to 
distinguish them more clearly from produc¬ 
tions. 

Intuitively, a set of items can be used to 
represent a stage in the parsing process; for 
example, the item 

[A —* a . 0] 

indicates that an input string derivable from 
a has just been seen, and, if we next see an 
input string derivable from 0, we may be 
able to reduce by the production A —* a0. 

Suppose the portion of the input that we 
have seen to this point has been reduced to 
the viable prefix ya. Then the item [A —» 
a . 0\ is said to be valid for yet if yA is also a 
viable prefix. In general, more than one item 
is valid for a given viable prefix; the set of 
all items which are valid at a particular 

* Some authors have used the term “confieura- 
tion” for item. 
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stage of the parse corresponds to the current 
state of the parser. 

As an example, let us examine the viable 
prefix 

LIST 7 

in G\. The item 

[LIST -» LIST . ELEMENT] 

is valid for this prefix, since, setting y to the 
empty string and a to LIST S’ in the defini¬ 
tion above, we see that 7 LIST (which is 
just LIST) is a viable prefix. In other words, 
when this item is valid, we have seen a por¬ 
tion of the input that can be reduced to the 
viable prefix, and we expect to see next a 
portion of the input that can be reduced to 
ELEMENT. 

The item 

[LIST — . ELEMENT] 

is not valid for LIST S’ however, since 
setting 7 to LIST S’ and a to the empty 
string we obtain 

LIST S’ LIST 

which is not a viable prefix. 

The reader can (and should) verify that 
the state corresponding to the viable prefix 
LIST S’ is associated with the set of items: 

[LIST — LIST S’ • ELEMENT] 
[ELEMENT — . ‘a’] 

[ELEMENT — . ‘ b’\ 

If 7 is a viable prefix, we shall use V(y) to 
denote the set of items that are valid for 7. If 
7 is not a viable prefix, 7(7) will be empty. 
We shall associate a state of the parser with 
each set of valid items and construct the 
entries in the parsing action for that state 
from the set of items. There is a finite num¬ 
ber of productions, thus only a finite number 
of items, and thus a finite number of possi¬ 
ble states associated with every grammar G. 

6.2 Constructing the Collection of Accessible 
Sets of Items 

We shall now describe a constructive pro¬ 
cedure for generating all of the states and, 
at the same time, generating the parsing 
action and goto table. As a running ex¬ 


ample, we shall construct parsing action and 
goto tables for Gi. 

First, we augment the grammar with the 
production 

ACCEPT LIST 

where in general LIST would be the start 
symbol of the grammar (here Gi). A reduc¬ 
tion by this production corresponds to the 
accept action by the parser. 

Next we construct Io = 7(“’), the set of 
items valid for the viable prefix consisting 
of the empty string. By definition, for G\ this 
set must contain the item 

[ACCEPT — . LIST] 

The dot in front of the nonterminal LIST 
means that, at this point, we can expect to 
find as the remaining input any sentence 
derivable from LIST. Thus, J 0 must also 
contain the two items 

[LIST -» . LIST ’,’ ELEMENT] 
[LIST -> . ELEMENT] 

obtained from the two productions for the 
nonterminal LIST. The second of the items 
has a dot in front of the nonterminal ELE¬ 
MENT, so we should also add to the initial 
state the items 

[ELEMENT — . ‘a’] 
[ELEMENT -» . V] 

corresponding to the two productions for 
element. These five items constitute Io. 
We shall associate state 0 with 7 0 . 

Now suppose that we have computed 
V(y), the set of items which are valid for 
some viable prefix 7 . Let A be a terminal or 
nonterminal symbol. We compute V(yX) 
from F(y) as follows: 

(1) For each item of the form [A —* 
a . X0] in 7 ( 7 ), we add to V(yX) 
the item [A —» aX . 0 ]. 

(2) We compute the closure of the set of 
items in V(yX); that is, for each item 
of the form [B —* a . C/8] in V(yX), 
where C is a nonterminal symbol, we 
add to V(yX) the items 

[C - . «i] 

[C -» . at] 

[C-‘. «n] 
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where C -» a h • • • , C -* a„ are all 
the productions in G with C on the 
left side. If one of these items is al¬ 
ready in V(yX) we do not duplicate 
this item. We continue to apply this 
process until no new items can be 
added to V(yX). 

It can be shown that steps (1) and (2) 
compute exactly the items that are valid for 
yX [Aho and Ullman (1972a)]. 

For example, let us compute h = 
F(LIST), the set of items that are valid for 
the viable prefix LIST. We apply the above 
construction with y = " and X = LIST, and 
use the five items in 7 0 . 

In step (1) of the above construction, we 
add the items 

[ACCEPT -*• LIST .] 

[LIST — LIST . y ELEMENT] 

to h. Since no item in h has a nonterminal 
symbol immediately to the right of the dot, 
the closure operation adds no new items to 
/i. The reader should verify that these two 
items are the only items valid for the viable 
prefix. We shall associate state 1 with h. 

Notice that the above construction is com¬ 
pletely independent of y ; it needs only the 
items in F(y), and X. For every set of items 
I and every grammar symbol X the above 
construction builds a new set of items which 
we shall call GOTO(7, X); this is essentially 
the same goto function encountered in the 
last two sections. Thus, in our example, we 
have computed 


GOTO(7„, LIST) = 7i 

We can extend this GOTO function to 
strings of grammar symbols as follows: 

GOTO(7, *’) = 7 


GOTO (7, yX) = GOTO (GOTO (7, y), X) 

where 7 is a string of grammar symbols and 
A" is a nonterminal or terminal symbol. If 
I = V(a), then 7 = GOTO(7„, a). Thus 
GOTO(7 0 , a) 5* <p if and only if a is a viable 
prefix, where 7 0 = F("). 

The sets of items which can be obtained 
from 7 0 by GOTO’s are called the accessible 
sets of items. We build up the set of accessi¬ 


ble sets of items by computing GOTO(7, X), 
for all accessible sets of items 7 and gram¬ 
mar symbols X; whenever the GOTO con¬ 
struction comes up with a new nonempty set 
of items, this set of items is added to the set 
of accessible sets of items and the process 
continues. Since the number of sets of items 
is finite, the process eventually terminates. 

The order in which the sets of items are 
computed does not matter, nor does the 
name given to each set of items. We will 
name the sets of items 7 0 , h, 7j, • • • in the 
order in which we create them. We shall 
then associate state i with 7,. 

Let us return to G\. We have computed 
7 0 , which contained the items 

[ACCEPT . LIST] 

[LIST — . LIST y ELEMENT] 

[LIST — . ELEMENT] 

[ELEMENT -» . ‘a’] 

[ELEMENT -* . ‘6’] 

We now wish to compute GOTO(7 0 , X) for 
all grammar symbols X. We have already 
computed 

GOTO(7 0 , LIST) - h 
To determine GOTO(7 0 , ELEMENT), we 
look for all items in 7 0 with a dot immedi¬ 
ately before ELEMENT. We then take 
these items and move the dot to the right of 
ELEMENT. We obtain the single item 

[LIST -» ELEMENT .] 

The closure operation yields no new items 
since this item has no nonterminal to the 
right of the dot. We call the set with this 
item It. Continuing in this fashion we find 
that: 

GOTO(7 0 , ‘a’) contains only 
[ELEMENT — ‘o’ .] 

GOTO(7 0 , ‘b’) contains only 
[ELEMENT — ‘6’ .] 

and GOTO(7 0 , 7) and GOTO(7 0 , T) are 
empty. Let us call the two nonempty sets 
h and 7 4 . We have now computed all sets of 
items that are directly accessible from 7 0 . 

We now compute all sets of items that are 
accessible from the sets of items just com¬ 
puted. We continue computing accessible 
sets of items until no more new sets of items 
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are found. The following table shows the 
collection of accessible sets of items for Gf 

If. [ACCEPT -> . LIST] 

[LIST — . LIST 7 ELEMENT] 
[LIST -* . ELEMENT] 
[ELEMENT — . ‘o’] 

[ELEMENT . ‘b’] 


If [ACCEPT LIST .] 

[LIST LIST . 7 ELEMENT] 

If [LIST ELEMENT .] 

[ELEMENT -> ‘a’ .] 

If [ELEMENT — ‘6’ .] 

It: [LIST -> LIST 7 . ELEMENT] 
[ELEMENT — . ‘a’] 

[ELEMENT — . V] 

If [LIST -» LIST 7 ELEMENT .] 

The GOTO function on this collection can 
be portrayed as a directed graph in which 
the nodes are labeled by the sets of items 
and the edges by grammar symbols, as fol¬ 
lows: 



Here, we used i in place of /<. 

For example, we observe 

GOTO(0, ") = 0 

GOTO(0, LIST 7) = 5 

GOTO(0, LIST 7 ELEMENT) = 6 

Observe that there is a path from vertex 0 
to a given node if and only if that path spells 
out a viable prefix. Thus, GOTO(0, ‘ab’) is 
empty, since ‘ab’ is not a viable prefix. 


6.3 Constructing the Parsing Action and Goto 
Tables from the Collection of Sets of Items 

The parsing action table is constructed 
from the collection of accessible sets of items. 
From the items in each set of items I. we 
generate parsing actions. An item of the 
form 


[A — * a . ‘a’ /3] 

in I, generates the parsing action 
if (input = ‘a’) shift t 

where GOTO(/„ ‘a’) - I t . 

An item with the dot at the right end of 
the production is called a completed item. A 
completed item [A —* a .] indicates that we 
may reduce by production A —* a. However, 
with an LR(1) parser we must determine 
on what input symbols this reduction is 
possible. If ‘ai’, ‘at, • • • , ‘a„’ are these 
symbols and ‘a\, ‘at, • • • , ‘a n ’ are not asso¬ 
ciated with shift or accept actions, then we 
would generate the sequence of parsing ac¬ 
tions: 

if(input = ‘ai’) reduce by: A —* a 
if (input = ‘at) reduce by: A —* a 

if(input = ‘a,’) reduce by: A —* a 

As we mentioned in the last section, if the 
set of items contains only one completed 
item, we can replace this sequence of parsing 
actions by the default reduce action 

reduce by: A —* a 

This parsing action is placed after all shift 
and accept actions generated by this set of 
items. 

If a set of items contains more than one 
completed item, then we must generate 
conditional reduce actions for all completed 
items except one. In a while we shall ex¬ 
plain how to compute the set of input sym¬ 
bols on which a given reduction is permissi¬ 
ble. 

If a completed item is of the form 
[ACCEPT —»<S .] 
then we generate the accept action 
if(input = *$’) accept 
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where *$’ is the right endmarker for the input 
string. 

Finally, if a set of items generates no re¬ 
duce action, we generate the default error 
statement. This statement is placed after 
all shift and accept actions generated from 
the set of items. 

Returning to our example for Gi, from 
To we would generate the parsing actions: 

if(input = ‘o’) shift 3 
if(input = V) shift 4 
error 

Notice that these are exactly the same pars¬ 
ing actions as those for state 0 in the parser 
of Section 4. Similarly, / 3 generates the ac¬ 
tion 

reduce by: ELEMENT — > ‘o’ 

The goto table is used to compute the new 
state after a reduction. For example, when 
the reduction in state 3 is performed we al¬ 
ways have state 0 to the left of ‘o’. The new 
state is determined by simply noting that 

GOTO(/„, ELEMENT) - J, 

This gives rise to the code 

if (state = 0) goto = 2 

for ELEMENT in the goto table. 

In general, if nonterminal A has precisely 
the following GOTO’s in the GOTO graph: 

GOTO(/. t , A) = /«, 

GOTO(/ m , A) = I tt 

GOTO A) = I tn 

then we would generate the following repre¬ 
sentation for column A of the goto table: 

A: if(state = Si) goto = t x 
if(state = Si) goto = h 

if(state = s n ) goto = t n 

Thus, the goto table is simply a representa¬ 
tion of the GOTO function of the last sec¬ 
tion, applied to the nonterminal symbols. 

We must now determine the input sym¬ 
bols on which each reduction is applicable. 
This will enable us to detect ambiguities and 
difficult-to-parse constructs in the grammar, 


and to decide between reductions if more 
than one is possible in a given state. In 
general, this is a complex task; the most 
general solution of this problem was given by 
[Knuth (1965)], but his algorithm suffers 
from large time and memory requirements. 
Several simplifications have been proposed, 
notably by [DeRemer (1969 and 1971)], 
which lack the full generality of Knuth’s 
technique, but can construct practical par¬ 
sers in reasonable time for a large class of 
languages. We shall describe an algorithm 
that is a simplification of Knuth’s algorithm 
which resolves all conflicts that can be re¬ 
solved when the parser has the states as 
given above. 

6.4 Computing Lookahead Sets 

Suppose [A —» a . 0 ] is an item that is 
valid for some viable prefix 7 a. We say that 
input symbol ‘a’ is applicable for [A —► a . 0 ] 
if, for some string of terminals ‘w’, both 
yatfaw’ and yA'aw’ are right sentential 
forms. The right endmarker '$’ is applicable 
for [A —> a . 0\ if both ya$ and 7 A are 
right sentential forms. 

This definition has a simple intuitive ex¬ 
planation when we consider completed items. 
Suppose input symbol ‘a’ is applicable for 
completed item [A —> a .]. If an LR(1) 
parser makes the reduction specified by this 
item on the applicable input symbol ‘a’, 
then the parser will be able to make at least 
one more shift move without encountering 
an error. 

The set of symbols that are applicable for 
each item will be called the lookahead set 
for that item. From now on we shall in¬ 
clude the lookahead set as part of an item. 
The production with the dot somewhere in 
the right side will be called the core of the 
item. For example, 

([ELEMENT —* ‘a’ .], {',’, ‘$’)) 
is an item of Gi with core 

[ELEMENT ‘o’ .] 

and lookahead set {“,’, 

We shall now describe an algorithm that 
will compute the sets of valid items for a 
grammar where the items include their 
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lookahead sets. Recall that in the last sec¬ 
tion items in a set of items arose in two ways: 
by goto calculations, and then by the closure 
operation. The first type of calculation is 
very simple; if we have an item of the form 

(W -> « • m L) 

where X is a grammar symbol and L is a 
lookahead set, then when we perform the 
goto operation on X on this item, we obtain 
the item 

([A - <xX . 0], L) 

(i.e., the lookahead set is unchanged). 

It is somewhat harder to compute the 
lookahead sets in the closure operation. 
Suppose there is an item of the form 

([A -a. 50], L) 

in a set of items, where 5 is a nonterminal 
symbol. We must add items of the form 

([5 - . a], U) 

where B —> 5 is some production in the 
grammar. The new lookahead set L' will 
contain all terminal symbols which are the 
first symbol of some sentence derivable from 
any string of the form 0 ‘a’, where ‘a’ is a 
symbol in L. 

If, in the course of carrying out this con¬ 
struction, a set of items is seen to contain 
items with the same core; e.g., 

([A — oi . 0], Za) 
and ([A —* at . 0], Li) 

then these items are merged to create a sin¬ 
gle item; e.g., ([A — a- . 0], L x (J Li). 

We shall now describe the algorithm for 
constructing the collection of sets of items 
in more detail by constructing the valid sets 
of items for grammar Gi. Initially, we con¬ 
struct I 0 by starting with the single item 

([ACCEPT — . LIST], {“$’)) 

We then compute the closure of this set of 
items. The two productions for LIST give 
rise to the two items 

([LIST — . LIST 7 ELEMENT], {‘$’|) 
and ([LIST —. ELEMENT], {“$’ J) 

The first of these two items gives rise, 


through the closure operation, to two addi¬ 
tional items 

([LIST — . LIST 7 ELEMENT], {','}) 
and ([LIST — . ELEMENT], {‘,’)) 

since the first terminal symbol of any string 
derivable from 

7 ELEMENT 

is always Since all items with the same 
core are merged into a single item with the 
same core and the union of the lookahead 
sets, we currently have the following items 
in 7 0 : 

([ACCEPT — . LIST], {‘$’}) 

([LIST — . LIST 7 ELEMENT], [7, “$’)) 
([LIST — . ELEMENT], [7, *$’}) 

The first two of these items no longer give 
rise to any new items when the closure 
operation is applied. The third item gives 
rise to the two new items: 

([ELEMENT —* . ‘a’], [7, <$’)) 
([ELEMENT — . ‘b’], {*,’, “$’]) 

and these five items make up I 0 . 

We shall now compute 

It = GOTO (To, ‘a’). 

First we add the item 

([ELEMENT — ‘a’ .], “$’)) 

to It, since ‘a’ appears to the right of the 
dot of one item in /<>. The closure operation 
adds no new items to It. 

It contains a completed item. The look¬ 
ahead set “$’) tells us on which input 
symbols the reduction is applicable. 

The reader should verify that the com¬ 
plete collection of sets of items for G\ is: 


[ACCEPT . LIST], 

m 

[LIST - . LIST 7 ELEMENT], 

(7, ‘$’] 

[LIST — . ELEMENT], 

17. ‘$’1 

[ELEMENT — . ‘a’], 

17. ‘S’! 

[ELEMENT . ‘6’], 

17. ‘S’! 

[ACCEPT — LIST .], 

m 

[LIST LIST . ELEMENT], 

17. ‘$’1 

[LIST - ELEMENT .], 

1*,’. ‘S’! 

[ELEMENT - V .], 

(7. ‘S’] 
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[ELEMENT -• ‘V .], [ 

v, r\ 

U: 

[LIST - LIST 7 . ELEMENT], ( 

7, T) 


[ELEMENT - . ‘a’], | 

7, ‘S’! 


[ELEMENT - . ‘6’], [ 

7, ‘S’l 

/«: 

[LIST - LIST 7 ELEMENT .], [ 

7, ‘S’) 


Although the situation does not occur 
here, if we generate a set of items 1 1 such that 
It has the same set of cores as some other 
set of items I, already generated, but 7, ^ 
It, then we combine 7, and I, into a new set 
of items 7 by merging the lookahead sets of 
items with the same cores. We must then 
compute GOTO(7, X) for all grammar sym¬ 
bols X. 

The lookahead sets on the completed 
items give the terminal symbols for which 
the reductions should be performed. There 
is a possibility that there are ambiguities in 
the grammar, or the grammar is too complex 
to allow a parser to be constructed by this 
technique; this causes conflicts to be dis¬ 
covered in the actions of the parser. For ex¬ 
ample, suppose there is a set of items 7, in 
which ‘a* gives rise to the parsing action 
shift because GOTO(7„ ‘a') exists. Suppose 
also that there is a completed item 
(IA-»«.],L) 

in 7„ and that the terminal symbol ‘o’ is in 
the lookahead set L. Then we have no way 
of knowing which action is correct in state s 
when we see an ‘o’; we may shift ‘o’, or we 
may reduce by A —* a. Our only recourse is 
to report a shift-reduce conflict. 

In the same way, if there are two reduc¬ 
tions possible in a state because two com¬ 
pleted items contain the same terminal sym¬ 
bol in their lookahead sets, then we cannot 
tell which reduction we should do; we must 
report a reduce-reduce conflict. 

Instead of reporting a conflict we may 
attempt to proceed by carrying out all con¬ 
flicting parsing actions, either by parallel 
simulation [Earley (1970)] or by backtrack¬ 
ing [Pager (1972b)]. 

A set of items is consistent or adequate if it 
does not generate any shift-reduce or reduce- 
reduce conflicts. A collection of sets of items 
is valid if all its sets of items are consistent; 
our collection of sets of items for G\ is valid. 

We summarize the parsing action and goto 


table construction process: 

(1) Given a grammar G, augment the 
grammar with a new initial produc¬ 
tion 

ACCEPT — S 

where S is the start symbol of G. 

(2) Let 7 be the set with the one item 

([ACCEPT —► . 5], {'$’}) 

Let 7o be the closure of 7. 

(3) Let C, the current collection of ac¬ 
cessible sets of items, initially contain 
only 7 0 . 

(4) For each 7 in C, and for each grammar 
symbol X, compute 7' = GOTO (7, X). 
Three cases can occur: 

a. 7' = I" for some 7" already in C. 
In this case, do nothing. 

b. If the set of cores of 7' is distinct 
from the set of cores of a set of 
items already in C, then add I' to C. 

c. If the set of cores of 7' is the same 

as the set of cores of some 7" al¬ 
ready in A but I r I ", then let 

1" be the set of items 

([A-> a. j8], Lx U Lt) 

such that 

([A —* a . 0], 7a) is in I' and 
([A —► a . 0], U) is in I". 

Replace I" by I m in C. 

(5) Repeat step 4 until no new sets of 
items can be added to C. C is called 
the LALR(l) collection of sets of items 
for G. 

(6) From C try to construct the parsing 
action and goto tables as in Section 
6.3. 

If this technique succeeds in producing a 
collection of sets of items for a given gram¬ 
mar in which all sets of items are consistent, 
then that grammar is said to be an LALR(l) 
grammar. LALR(l) grammars include many 
important classes of grammars, including 
the LL(1) grammars [Lewis and Stearns 
(1968)], the simple mixed strategy prece¬ 
dence grammars [McKeeman, Horning, and 
Wortman (1970)], and those parsable by 
operator precedence techniques. Techniques 
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for proving these inclusions can be found in 
[Aho and Ullman (1972a and 1973a)]. 

Step (4) can be rather time-consuming to 
implement. A simpler, but less general, 
approach would be to proceed as follows. Let 
FOLLOW(A) be the set of terminal symbols 
that can follow nonterminal symbol A in a 
sentential form. If A can be the rightmost 
symbol of a sentential form, then '$’ is in¬ 
cluded in FOLLOW(A). We can compute the 
sets of items without lookaheads as in Section 
6.2. Then in each completed item [A —>• a .] 
we can approximate the lookahead set L for 
this item by FOLLOW(A) (In general, L is 
a subset of FOLLOW(A).) The resulting 
collection of sets of items is called the 
SLR(l) collection. If all sets of items in the 
SLR(l) collection are consistent, then the 
grammar is said to be simple LR(1) [De- 
Remer (1971)]. Although not every LALR(l) 
grammar is simple LR(1), every language 
generated by an LALR(l) grammar is also 
generated by a simple LR(1) grammar 
([Aho and Ullman (1973a)] contains more 
details). 


7. PARSING AMBIGUOUS GRAMMARS 

It is undesirable to have undetected ambigui¬ 
ties in the definition of a programming 
language. However, an ambiguous grammar 
can often be used to specify certain language 
constructs more easily than an equivalent 
unambiguous grammar. We shall also see 
that we can construct more efficient parsers 
directly from certain ambiguous grammars 
than from equivalent unambiguous gram¬ 
mars. 

If we attempt to construct a parser for 
an ambiguous grammar, the LALR(l) 
parser construction technique will generate 
at least one inconsistent set of items. Thus, 
the parser generation technique can be used 
to determine that a grammar is unambigu¬ 
ous. That is to say, if no inconsistent sets of 
items are generated, the grammar is guaran¬ 
teed to be unambiguous. However, if an 
inconsistent set of items is produced, then 
all we can conclude is that the grammar is 
not LALR(l). The grammar may or may 
not be ambiguous. (There is no general 


algorithm to determine if a context-free 
grammar is ambiguous (see, for example 
[Aho and Ullman (1972a)]). 

Inconsistent sets of items are useful in 
pinpointing difficult-to-parse or ambiguous 
constructions in a given grammar. For 
example, a production of the form 

A —► AA 

in any grammar will make that grammar 
ambiguous and cause a parsing action con¬ 
flict to arise from sets of items containing 
the items with the cores 

[A — A A .] 

[A — A . A] 

Constructions which are sufficiently com¬ 
plex to require more than one symbol of 
lookahead also result in parsing action con¬ 
flicts. For example, the grammar 

A‘a’ 

A —* ‘a’ | *’ 

is an LALR(2) but not LALR(l) grammar. 

Experience with an LALR(l) parser 
generator called YACC at Bell Laboratories 
has shown that a few iterations with the 
parser generator are usually sufficient to re¬ 
solve the conflicts in an LALR(l) collec¬ 
tion of sets of items for a reasonable pro¬ 
gramming language. 

Example 7.1: Consider the following pro¬ 
ductions for “if-then” and “if-then-else” 
statements: 

S —* ‘if b then’ S 
S —► 'if b then’ S ‘else ’S 

If these two productions appear in a gram¬ 
mar, then that grammar will be ambiguous; 
the string 

‘if b then if b then’ 5 ‘else’ <S 
can be parsed in two ways as shown: 
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In most programming languages, the first 
phrasing is preferred. That is, each new 
‘else’ is to be associated with the closest 
“unelsed” ‘then’. 

A grammar using these ambiguous produc¬ 
tions to specify if-then-else statements will 
be smaller and, we feel, easier to compre¬ 
hend than an equivalent unambiguous 
grammar. In addition if a grammar has only 
ambiguities of this type, then we can con¬ 
struct a 'alid LALR(l) parser for the gram¬ 
mar merely by resolving each shift-reduce 
conflict in favor of shift [Aho, Johnson, and 
Ullman (1973)]. 

Example 7.2: Consider the ambiguous 
grammar* 


[5 — ‘if b then’ S ‘else’ . 3], 1‘else’, ‘$’1 

[S —* . ‘if b then’ S], (‘else’, ‘S’} 

[3 — . ‘if 6 then’ S ‘else’ 5], (‘else’, ‘S’ 1 

[S-.'a’l, (‘else’,‘$’1 

It: [S -* ‘if 6 then’ S ‘else’ S .], (‘else’, ‘VI 

/ 4 contains a shift-reduce conflict. On the 
input ‘else’, Z 4 says that either a shift move 
to 7 S is permissible, or a reduction by pro¬ 
duction 

S —► ‘if b then’ 5 

is possible. If we choose to shift, we shall 
associate the incoming ‘else’ with the last 
unelsed ‘then’. This is evident because the 
item with the core 

[5 —*• ‘if b then ’S . ‘else’ 5] 

in / 4 gives rise to the shift action. 

The complete parsing action table, with 
the conflict resolved, and the goto table con¬ 
structed from this collection of sets of items 
are shown below: 


S —► ‘if 6 then ’S 
S —► ‘if b then ’S ‘else ’S 
S —» ‘a’ 

in which each else is to be associated with 
the last unelsed ‘then’. The LALR(l) col¬ 
lection of sets of items for this grammar is as 
follows: 

If. [ACCEPT — • SI, |‘$>| 

[5 —► . ‘if 6 then’ S], |‘S’| 

(3 -* . ‘if b then’ S ‘else’ S], j‘$’ 

[5 — . ‘a’l, (‘VI 

h: (ACCEPT — 3.], |‘S’| 

If. [3 — ‘if b then’ . 31, (‘else’, ‘$’| 

[S — ‘if b then’ . S ‘else’ 3], (‘else’, ‘S’ } 

(3 — . ‘if 6 then’ 3], (‘else’, ‘S’ | 

[5 — . ‘if 6 then’ 3 ‘else’ S], (‘else’, ‘S’ ! 

[3 — . ‘a’], (‘else’,‘$’1 


Parsing Action Table 

0: if (input = ‘if b then’) shift 2 
if(input * ‘a’) shift 3 
error 

1 : if(input = $) accept 
error 

2: if(input = ‘if b then’) shift 2 
if(input = ‘a’) shift 3 
error 

3: reduce by: S —*• ‘a’ 

4: if(input = ‘else’) shift 5 
reduce by: S —* ‘if b then’ S 

5: if(input = ‘if b then’) shift 2 
if(input = ‘a’) shift 3 
error 

6: reduce by: S —» ‘if b then ’S ‘else ’S 


If. [3—‘a’.], (‘else’,‘$’| 

I*. [3 — ‘if b then’ S .], (‘else’, ‘$’| 

[3 — ‘if 6 then’ 5 . ‘else’ S], (‘else’, ‘S’) 


* The following grammar is an equivalent unam¬ 
biguous grammar: 

S — ‘if 6 then' S 
S — ‘if b then’ 3i ‘else’ 3 

3! — ‘if b then’ Si ‘else’ Si 
Si - ‘a’ 


Goto Table 

S: if(state = 0) goto = 1 
if(state = 2) goto = 4 
goto = 6 

Given an ambiguous grammar, with the 
appropriate rules for resolving the ambigui¬ 
ties we can often directly produce a smaller 
parser from the ambiguous grammar than 
from the equivalent unambiguous grammar. 
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However, some of the “optimizations” dis¬ 
cussed in the next section will make the par¬ 
ser for the unambiguous grammar as small 
as that for the ambiguous grammar. 

Example 7.3: Consider the following gram¬ 
mar G 3 for arithmetic expressions: 

E-*E‘+’ E 

E —► E E 

e — ‘(’Ey 

E —*■ ‘a’ 

where ‘a’ stands for any identifier. Assuming 
that + and * are both left associative and 
* has higher precedence than +, there are 
two things wrong with this grammar. First, 
it is ambiguous in that the operands of the 
binary operators ‘+’ and V can be associ¬ 
ated in any arbitrary way. For example, 
‘a + aa’ can be parsed as 



or as 



The first parsing gives the usual left-to-right 
associativity, the second a right-to-left 
associativity. 

If we rewrote the grammar as G t : 

E-*E‘+’ T 
E —* E T 
E —* T 
T -> ‘(’Ey 
T —» ‘a’ 

then we would have eliminated this am¬ 
biguity by imposing the normal left-to-right 
associativity for + and *. However, this 
new grammar has still one more defect; + 
and * have the same precedence, so that an 
expression of the form ‘a+a*a’ would be 
evaluated as (a+a)*a. To eliminate this, 
we must further rewrite the grammar as 
G s : 

E —* E *+’ T 
E —* T 
T -* T V F 
T —> F 
F - ‘(’Ey 
F —» ‘a’ 

We can now construct a parser for (?* 
quite easily, and find that we have 12 states; 
if we count the number of parsing actions in 
the parser (i.e., the sum of the number of 
shift and reduce actions in all states to¬ 
gether with the goto actions) we see that the 
parser for G s has 35 actions. 

In contrast, the parser for G 3 has only 10 
states, and 29 actions. A considerable part 
of the saving comes from the elimination of 
the nonterminals T and F from (? 5 , as well as 
the elimination of the productions E —* T 
and T —> F. 

Let us discuss the resolution of parsing 
action conflicts in G 3 in somewhat more de¬ 
tail. There are two sets of items in the 
LALR(l) collection of sets of items for G 3 
which generate conflicts in their parsing ac¬ 
tions: 

[E-*E . l +’E], {*+*, V, ‘)’, ‘$’\ 
[E^E.‘*’E], (•+’, v, y, T! 

[E-*£•+• E .], {*+’, V, y, T} 
and [E—*E . E\, {*+’, *)’, 

[e-*e.‘*’e\, {“+’, v, y, '$’} 

[E^E‘*’E.], V, y, '$’} 
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In both sets of items, shift-reduce conflicts 
arise on the two terminal symbols '+' and 
V. For example, in the first set of items on 
an input of “+’ we may generate either a 
reduce action or a shift action. Since we wish 
+ to be left associative, we wish to reduce 
on this input; a shift would have the effect of 
delaying the reduction until more of the 
string had been read, and would imply right 
associativity. On the input symbol V, how¬ 
ever, if we did the reduction we would end 
up parsing the string l a+a*a’ as ( a+a)*a ; 
that is, we would not give * higher prece¬ 
dence than +. Thus, it is correct to shif t, on 
this input. Using similar reasoning, we see 
that it is always correct to generate a re¬ 
duce action from the second set of items; on 
the input symbol V this is a result of the 
left associativity of *, while on the input 
symbol ‘+’ this reflects the precedence rela¬ 
tion between + and *. 

We conclude this section with an example 
of how this reasoning can be applied to our 
grammar G\. We noted earlier that the 
grammar <j S : 

LIST -* LIST 7 LIST 
LIST — ‘a’ 

LIST — V 

is ambiguous, but this ambiguity should no 
longer be of concern. Assuming that the 
language designer wants to treat 7 as a left 
associative operator, then we can produce a 
parser which is smaller and faster than the 
parser for G\ produced in the last section. 
The smaller parser looks like: 

Parsing Action Table 

0: if(input = ‘a’) shift 2 
if(input = ‘6’) shift 3 
error 

1: if (input = '$’) accept 
if (input = 7) shift 4 
error 

2: reduce by: LIST —» ‘a’ 

3: reduce by: LIST —* ‘b’ 

4: if (input = ‘a’) shift 2 
if(input = ‘6’) shift 3 
error 

5: reduce by: LIST -» LIST 7 LIST 


Goto Table 

LIST: if(state = 0) goto = 1 
goto = 5 

Notice that we have only 14 parsing ac¬ 
tions in this parser, compared to the 16 
which we had in the earlier parser for G\. In 
addition, the derivation trees produced by 
this parser are smaller since the nodes cor¬ 
responding to the nonterminal symbol ELE¬ 
MENT are no longer there. This in turn 
means that the parser makes fewer actions 
when parsing a given input string. Parsing 
of ambiguous grammars is discussed by 
[Aho, Johnson, and Ullman (1973)] in more 
detail. 


8. OPTIMIZATION OF LR PARSERS 

There are a number of ways of reducing the 
size and increasing the speed of an LR(1) 
parser without affecting its good error-de¬ 
tecting capability. In this section we shall 
list a few of many transformations that can 
be applied to the parsing action and goto 
tables of an LR(1) parser to reduce their 
size. The transformations we list are some 
simple ones that we have found to be effec¬ 
tive in practice. Many other transformations 
are possible and a number of these can be 
found in the references at the end of this 
section. 

8.1 Merging Identical States 

The simplest and most obvious “optimiza¬ 
tion” is to merge states with common parsing 
actions. For example, the parsing action 
table for G\ given in Section 5 contains 
identical actions in states 0 and 5. Thus, it is 
natural to represent this in the parser as: 

0: 5: if (input = ‘a’) shift 3 
if(input = ‘6’) shift 4 
error 

Clearly the behavior of the LR(1) parser 
using this new parsing action table is the 
same as that of the LR(1) parser using the 
old table. 
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8.2 Subsuming States 

A slight generalization of the transforma¬ 
tion in Section 8.1 is to eliminate a state 
whose parsing actions are a suffix of the 
actions of another state. We then label the 
beginning of the suffix by the eliminated 
state. For example, if we have: 

n: if (input = ‘x’) shift p 
if(input = V) shift q 
error 

and m: if(input = ‘y’) shift q 
error 

then we may eliminate state m by adding the 
label into the middle of state n: 

n: if (input = V) shift p 
m: if(input = ‘y’) shift q 
error 

Permuting the order of these statements 
can increase the applicability of this op¬ 
timization. (See Ichbiah and Morse (1970) 
for suggestions on the implementation of this 
optimization.) 

8.3 Elimination of Reductions by Single 

Productions 

A single production is one of the form 
A —+ X, where A is a nonterminal and X is 
a grammar symbol. If this production is not 
of any importance in the translation, then 
we say that the single production is se¬ 
mantically insignificant. A common situa¬ 
tion in which single productions arise occurs 
when a grammar is used to describe the 
precedence levels and associativities of 
operators (see grammar of Example 7.3). 
We can always cause an LR parser to avoid 
making these reductions; by doing so we 
make the LR parser faster, and reduce the 
number of states. (With some grammars, the 
size of the “optimized” form of the parsing 
action table may be greater than the un¬ 
optimized one.) 

We shall give an example in terms of G\ 
which contains the single production 

LIST — ELEMENT 

We shall eliminate reductions by this pro¬ 
duction from the parser for G\ found in Sec¬ 


tion 5. The only state which calls for a re¬ 
duction by this production is state 2. More¬ 
over, the only way in which we can get to 
state 2 is by the goto action 

ELEMENT: if(state = 0) goto = 2 

After the parser does the reduction in state 
2, it immediately refers to the goto action 

LIST: goto = 1 

at which time the current state becomes 1. 
Thus, the rightmost tree is only labeled with 
state 2 for a short period of time; state 2 
represents only a step on the way to state 1. 
We may eliminate this reduction by the sin¬ 
gle production by changing the goto action 
under ELEMENT to: 

ELEMENT: if(state = 0) goto = 1 

so that we bypass state 2 and go directly to 
state 1. We now find that state 2 can never 
be reached by any parsing action, so it can 
be eliminated. Moreover, it turns out here 
(and frequently in practice as well) that the 
goto actions for LIST and ELEMENT be¬ 
come compatible at this point; that is, the 
actions do not differ on the same state. It is 
always possible to merge compatible goto 
actions for nonterminals; the resulting parser 
has one less state, and one less goto action. 

Example 8.1: The following is a representa¬ 
tion of the parsing action and goto tables for 
an LR(1) parser for G\. It results from the 
parsing action and goto tables in Section 5 
by applying state merger (Section 8.1), and 
eliminating the reduction by the single pro¬ 
duction. 

Parsing Action Table 
0: 5: if (input = ‘a’) shift 3 
if (input = ‘6’) shift 4 
error 

1: if (input = “,’) shift 5 

if (input = $) accept 

3: reduce by: ELEMENT ‘a’ 

4: reduce by: ELEMENT -> ‘b’ 

6: reduce by: LIST - LIST 7 ELEMENT 

Goto Table 

LIST: ELEMENT: instate = 0) goto = 1 
goto = 6 
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These tables are identical with those for 
the ambiguous version of Gi, after the equal 
states have been identified. These tables 
differ only in that the nonterminal symbols 
LIST and ELEMENT have been explicitly 
merged in the ambiguous grammar, while 
the distinction is still nominally made in the 
tables above. 

In the general case, there may be a number 
of states which call for reductions by the 
same single production, and there may be 
other parsing actions in the states which call 
for these reductions. It is not always possi¬ 
ble, in general, to perform these modifica¬ 
tions without increasing the number of 
states; conditions which must be satisfied in 
order to profitably carry out this process 
are given in [Aho and Ullman (1973b)]. It 
is enough for our purposes to notice that if 
a reduction by a single production A —* X 
is to be eliminated, and if this reduction is 
generated by exactly one set of items con¬ 
taining the item with the core 

{A-+X.) 

then this single production can be eliminated. 
It turns out that the single productions 
which arise in the representation of operator 
precedence or associativity can always be 
eliminated; the result is typically the same 
as if an ambiguous grammar were written, 
and the conflicts resolved as discussed in 
Section 6. However, the ambiguous grammar 
generates the reduced parser immediately, 
without needing this optimizing algorithm 
[Aho, Johnson, and Ullman (1973)]. 

Other approaches to optimization of LR 
parsers are discussed by [Aho and Ullman 
(1972b)], [Anderson (1972)], [Jolliat (1973)], 
and [Pager (1970)]. [Anderson, Eve, and 
Homing (1973)], [Demers (1973)], and 
[Pager (1974)] also discuss the elimination of 
reductions by single productions. 


9. ERROR RECOVERY 

A properly designed LR parser will an¬ 
nounce that an error has occurred as soon 
as there is no way to make a valid continua¬ 
tion to the input already scanned. Un¬ 
fortunately, it is not always easy to decide 


what the parser should do when an error is 
detected; in general, this depends on the 
environment in which the parser is operating. 
Any scheme for error recovery must be 
carefully interfaced with the lexical analysis 
and code generation phases of compilation, 
since these operations typically have “side 
effects” which must be undone before the 
error can be considered corrected. In addi¬ 
tion, a compiler should recover gracefully 
from each error encountered so that subse¬ 
quent errors can also be detected. 

LR parsers can accommodate a wide 
variety of error recovery stratagems. In 
place of each error entry in each state, we 
may insert an error correction routine which 
is prepared to take some extraordinary ac¬ 
tions to correct the error. The description of 
the state as given by the set of items fre¬ 
quently provides enough context information 
to allow for the construction of sophisticated 
error recovery routines. 

We shall illustrate one simple method by 
which error recovery can be introduced into 
the parsing process. This method is only one 
of many possible techniques. We introduce 
error recovery productions of the form 

A -> error 

into the grammar for certain selected non¬ 
terminals. Here, error is a special terminal 
symbol. These error recovery productions 
will introduce items with cores of the form 

[A —» . error] 

into certain states, as well as introducing 
new states of the form 

[A —* error .] 

When the LR parser encounters an error, it 
can announce error and replace the current 
input symbol by the special terminal symbol 
error. The parser can then discard trees 
from the parse forest, one at a time from 
right-to-left, until the current state (the 
state on the rightmost tree in the parse 
forest) has a parsing action shift on the in¬ 
put error. The parser has now reached a 
state with at least one item of the form 

[A —> . error] 

The parser can then perform the shift 
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action and reduce by one of the error re¬ 
covery productions 

A —> error 

(If more than one error recovery production 
is present, a choice would have to be speci¬ 
fied.) On reducing, the parser can perform a 
hand-tailored action associated with this 
error situation. One such action could be to 
skip forward on the input until an input 
symbol ‘a’ was found such that ‘a’ can 
legitimately occur either as the last symbol 
of a string generated by A or as the first 
symbol of a string that can follow A. 

Certain automatic error recovery actions 
are also possible. For example, the error re¬ 
covery productions can be mechanically 
generated for any specified set of nontermi¬ 
nals. Parsing and error recovery can proceed 
as above, except that on reducing by an error 
recovery production, the parser can auto¬ 
matically discard input symbols until it finds 
an input symbol, say ‘o’, on which it can 
make a legitimate parsing action, at which 
time normal parsing resumes. This would 
correspond to assuming that an error was 
encountered while the parser was looking for 
a phrase that could be reduced to nontermi¬ 
nal A. The parser would then assume that 
by skipping forward on the input to the 
symbol ‘a’ it would have found an instance 
of nonterminal A. 

Certain error recovery schemes can pro¬ 
duce an avalanche of error messages. To 
avoid a succession of error messages stem¬ 
ming from an inappropriate recovery, a 
parser might suppress the announcement of 
subsequent errors until a certain number of 
successful shift actions have occurred. 

We feel that, at present, there is no effi¬ 
cient general “solution” to the error re¬ 
covery problem in compiling. We see faults 
with any uniform approach, including the 
one above. Moreover, the success of any 
given approach can vary considerably from 
application to application. We feel that if a 
language is cleanly designed and well hu¬ 
man-engineered, automatic error recovery 
will be easier as well. 

Particular methods of error recovery dur¬ 
ing parsing are discussed by [Aho and Peter¬ 
son (1972)], [Graham and Rhodes (1973)], 


[James (1972)], [Leinius (1970)], [McGruther 
(1972)], [Peterson (1972)], and [Wirth 
(1968)]. 

10. OUTPUT 

In compiling, we are not interested in pars¬ 
ing but rather in producing a translation for 
the source program. LR parsing is eminently 
suitable for producing bottom-up transla¬ 
tions. 

Any translation which can be expressed 
as the concatenation of outputs which are 
associated with each production can be 
readily produced by an LR parser, without 
having to construct the forest representing 
the derivation tree. For example, we can 
specify a translation of arithmetic expressions 
from infix notation to postfix Polish notation 
in this way. To implement this class of trans¬ 
lations, when we reduce, we perform an 
output action associated with that produc¬ 
tion. For example, to produce postfix Polish 
from Gi, we can use the following transla¬ 
tion scheme: 

(1) E^E‘+’E 

(2) E- E E 

(3) E '(’£•)’ 

(4) E ‘a’ 

Here, as in Section 7, we assume that + 
and * are left associative, and that * has 
higher precedence than +. The translation 
element is the output string to be emitted 
when the associated reduction is done. Thus, 
if the input string 

‘a + a * (a + a)’ 
is parsed, the output will be 
‘aaaa + * +’ 

These parsers can also produce three ad¬ 
dress code or the parse tree as output with 
the same ease. However, more complex 
translations may require more elaborate 
intermediate storage. Mechanisms for im¬ 
plementing these translations are discussed 
in [Aho and Ullman (1973a)] and in [Lewis, 
Rosenkrantz, and Stearns (1973)]. It is our 
current belief that, if a complicated trans¬ 
lation is called for, the best way of imple- 
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menting it is by constructing a tree. Optimiz¬ 
ing transformations can then massage this 
tree before final code generation takes place. 
This scheme is simple and has low overhead 
when the input is in error. 


11. CONCLUDING REMARKS 

LR parsers belong to the class of shift-reduce 
parsing algorithms [Aho, Denning, and Ull- 
man (1972)]. These are parsers that operate 
by scanning their input from left-to-right, 
shifting input symbols onto a pushdown 
stack until the handle of the current right 
sentential form is on top of the stack; the 
handle is then reduced. This process is con¬ 
tinued either until all of the input has been 
scanned and the stack contains only the 
start symbol, or until an error has been en¬ 
countered. 

During the 1960s a number of shift-reduce 
parsing algorithms were found for various 
subclasses of the context-free grammars. The 
operator precedence grammars [Floyd 
(1963]), the simple precedence grammars 
[Wirth and Weber (1966)], the simple mixed 
strategy precedence grammars [McKeeman, 
Homing, and Wortman (1970)], and the 
uniquely invertible weak precedence gram¬ 
mars [Ichbiah and Morse (1970)] are some of 
these subclasses. The definitions of these 
classes of grammars and the associated 
parsing algorithms are discussed in detail in 
[Aho and Ullman (1972a)]. 

In 1965 Knuth defined a class of gram¬ 
mars which he called the LR(fc) grammars. 
These are the context-free grammars that 
one can naturally parse bottom-up using a 
deterministic pushdown automaton with 
fc-symbol lookahead to determine shift- 
reduce parsing actions. This class of gram¬ 
mars includes all of the other shift-reduce 
parsable grammars and admits of a parsing 
procedure that appears to be at least as effi¬ 
cient as the shift-reduce parsing algorithms 
given for these other classes of grammars. 
[Lalonde, Lee, and Homing (1971)] and 
[Anderson, Eve, and Horning (1973)] pro¬ 
vide some empirical comparisons between 
LR and precedence parsing that support 
this conclusion. 


In his paper Knuth outlined a method for 
constructing an LR parser for an LR gram¬ 
mar. However this algorithm results in 
parsers that are too large for practical use. 
A few years later [Korenjak (1969)] and par¬ 
ticularly [DeRemer (1969 and 1971)] suc¬ 
ceeded in substantially modifying Knuth’s 
original parser constructing procedure to 
produce parsers of practical size. Substan¬ 
tial progress has been made since in improv¬ 
ing the size and performance of LR parsers. 

The general theory of LR(fc) grammars 
and languages is developed in [Aho and Ull¬ 
man (1972a and 1973a)]. Proofs of the cor¬ 
rectness and efficacy of many of the con¬ 
structions in this paper can be found there. 

Perhaps the biggest advantage of LR 
parsing is that small, fast parsers can be 
mechanically generated for a large class of 
context-free grammars, that includes all 
other classes of grammars for which non¬ 
backtracking parsing algorithms can be 
mechanically generated. In addition, LR 
parsers are capable of detecting syntax errors 
at the earliest opportunity in a left-to-right 
scan of an input string, a property not en¬ 
joyed by many other parsing algorithms. 

Just as we can parse by constructing a 
derivation tree for an input string bottom-up 
(from the leaves to the root) we can also 
parse top-down by constructing the deriva¬ 
tion tree from the root to the leaves. A 
proper subclass of the LR grammars can 
be parsed deterministically top-down. These 
are the class of LL grammars, first studied 
by [Lewis and Stearns (1968)]. LL parsers 
are also efficient and have good error-de¬ 
tecting capabilities. In addition, an LL par¬ 
ser requires less initial optimization to be of 
practical size. However, the most serious 
disadvantage of LL techniques is that LL 
grammars tend to be unnatural and awk¬ 
ward to construct. Moreover, there are LR 
languages which do not possess any LL 
grammar. 

These considerations, together with prac¬ 
tical experience with an automatic parser 
generating system based on the principles 
expounded in this paper, lead us to believe 
that LR parsing is an important, practical 
tool for compiler design. 
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Syntax-Directed Specifications and Their Implementation 
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0. Overview 

1. Language Specification 

2. Attribute Grammars 

3. Syntax-Directed Specifications 

4. Implementation of Syntax-Directed Specifications 

5. Optimization of Storage 


6. Attribute Evaluators 



1. Language Specification 

1.1 conventional language specifications 

1.2 formal semantic methods 

denctational semantics 
ADA spedfication 

1.3 syntax-directed spedfication 

1.4 aspects of language spedfication 

generation of intermediate code 
static semantic analysis 

1.5 advantages of syntax-directed methods 

readability 

accuracy 

coverage 

portability 

automatability 
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2. Attribute Grammars 

2.1 basic definition 

attributes 
semantic rules 

2.2 synthesized attributes 

definition 

example 

2.3 inherited attributes 

definition 

example 

2.4 circularity 
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3. Syntax-Directed Specifications 

3.1 expression evaluation 

3.2 generation of parse trees 

3.3 static semantic checks 

3.4 environments 
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4. Implementation of Syntax-Directed Specifications 

4.1 dependencies 

4.2 1-attributed grammars 

4.3 implementation using bottom-up parsing 

4.4 implementation using top-down parsing 
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5. Optimization of Storage 


5.1 issues 

5.2 global attributes 

5.3 stack-like attributes 


5.4 evaluation order 
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6. Attribute Evaluators 

6.1 overview 

6.2 dependencies 

6.3 graph evaluators 

6.4 incremental evaluation 
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Code Generation Comprises Two Distinct Phases: 

1 . Compiler’s Front End Produces an 
Intermediate Representation (IR) 

Translation of the Source Program. 

. Translation is guided by the run-time 
semantics. 

. IR can be interpreted or translated to 
target code 

2. Compiler’s Code Generator translates IR 
into actual target machine code. 



IRs Can be High-Level or Low Level 

• A High-Level IR is closer to the source 
language. 

• The ultimate high-level IR is 
directly-executable encoding. 

. A Low-Level IR is closer to the target 
machine language. 

. The ultimate low-level IR is machine 
language. 

• The level of an IR is usually determined by 
the set of operators it employs. 
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IR and Target Code (TC) generation reflect a Run-Time 
model involving: 

1. Translation of data structures 

2. Memory allocation and management 

3. Implementation of Control Structures 

4. Procedure calls and Argument passing 

5. Design of an underlying Virtual Machine 
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Virtual Machine Organization 

. The Run-Time model is implemented in 
terms of a Virtual Machine. 

• The Virtual Machine’s operations are a 
blend of hardware instructions and software 
support routines. 

. Hardware instructions are used where 
applicable. 

• Support routines can be inline or ordinary 
closed subroutines. 



The Code Generator must: 

1. Efficiently implement primative and 
. structured data. 

2. Map data access paths to hardware 
addressing modes. 

3. Establish operand context and evaluation 

order A&+Q VS 

4. Allocate registers. 

5. Choose target machine instructions for IR 
code. 

6. Optionally provide Machine-Dependent and 
Peephole Optimizations. 



Code Generation IRs 

. To enhance retargetability, an IR may be 
translated to a lower-level Code 
Generation IR (CGIR). 

. Conventional IRs allow machine- 
independent optimization and 
communication between compiler passes. 

. A CGIR is designed for retargetable code 
generation. 

. If no IR optimization is performed, the CGIR 
may be generated directly. 

. Different source languages can share a 
common CGIR. They may not be able to 
share a common IR. 



Issues in CGIR design: 

, How easy is it to write a front-end 
translator? 

.. Can code generation be treated as a 
isolated package? 

. Efficiency of the code generation algorithm. 

. Can machine-independent optimizations be 
expressed in CGIR? 

. Is storage binding performed by the front- 
end or is it done by the back-end? 
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Machine Dependent Optimizations 

. Depend on the details of a particular 
architecture. 

. Normally can’t be discovered by an IR*level 
optimizer. 

.Typical machine-dependent optimizations 
are: 


1. Subsumption of more than one 
operation in the same instruction: 
Auto-increment/decrement, 
increment/test/conditional branch 
instruction,... 

2. Span-dependent branch selection 
[Syzmanski]. 

3. Strength-reduction. shift k minny 

4. Low-level CSEs (e.g., access-path 
computation). 

5. Redundant computations. 

£.<r. CoVDITtct/ CCD£S 
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Peephole Optimization 

. A Peephole Optimizer examines 
instructions in a small window, seeking to 
simplify them. 

. The window may comprise 2 or 3 physically 
adjacent instructions. 

Thus "BZ LI ;B L2; LI 
can be replaced with 
"BNZ L2; LI:". 

. If LOGICAL adjacency is used, even better 
optimizations are possible: 

"Jump L,..., L: Jump K" 
can be replaced with 
"Jump K.L: Jump K" 



Goals For a Retargetable Code Generator 

. Minimize machine-dependent modules 

. Provide reasonable speed and 
compactness of target code 

. Provide reasonable compilation speed 

Three approaches to retargetability may be noted: 

1. Interpretive code generation 

2. Pattern-matched code generation 

3. Table-driven code generation 



Interpretive code generation 

. CGIR is macro-expanded into real target 
code (P-Code is a well-known example). 

• For L languages and M machines, L front- 
ends plus M code generators are used. 

. Each code generator is an interpreter that 
translates virtual machine operations into 
target machine operations. 

. An improvement is U-Code [Sites] which 
maintains a state while interpreting CGIR 
instructions. 



Pattern Matching Code Generators 

. Idea is to separate machine description 
from the code generation algorithm. 

. Use pattern matching techniques to replace 
interpretation with case analysis. 

Techniques for pattern matching: 

. Exhaustive brute-force technique with 
dynamic programming [Aho, Ripken] 

. Knowledge-based rules that direct pattern 
matching [Fraser] 
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• Goal-directed heuristic search [Cattell], 
Create sub-goals as search continues; 
use heuristics to order subgoal selection, 
use heuristics to order patterns when trying 
to match. 
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Table-driven Code Generation 

. Automated enhancements of pattern- 
matched approaches. 

. A first phase reads in machine’s Instruction- 
Set Architecture and produces tables. 

. A second phase uses these tables and the 
CGIR to generate target code. 

. A lot of analysis that is normally carried out 
during code generation can be pre¬ 
compiled: 

1. Heuristic search can be done at 
Code-Generator-Generation time 
[PQCC]. 

2. Identify potentially looping and 
blocking configurations [Glanville]. 
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• A table of templates in conjunction with a 
template matching algorithm [Johnson] 

. Simple SLR parsing [Graham and Glanville] 


R1 -. + K R1 
R1 - + R1 K 

R1 - + t + K R2 R1 
R1 - + i + R2 K R1 
R1 - + R1 t + K R2 
R1 -. + R1 t + R2 K 

R1 - + R2 R1 
R1 -. + R1 R2 


{ADD R1 t = K} 
{ADD R1, = K} 

{ADD R1,K(R2)} 
{ADD R1,K(R2)} 
{ADD R1,K(R2)} 
{ADD R1,K(R2)} 

{ADD R1.R2} 
{ADD R1,R2} 
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. Attributed parsing [Ganapathi and Fischer] 

ByteiR - + BytetA ByteiR lsZero?iA 

ByteiR - + BytetA BytejR 

lsOne?iAA-Busy?iR 

EmitiINCBiR 

ByteiR - + ByteiA BytetR -Busy?iR 
EmitiADDB2iAiR 

ByteiR - + BytetA ByteiB 

GetTempiByteiR 

EmitiADDB3iAiBiR 
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Storage Binding 


Storage binding: mapping attributes in IR to machine' 

dependent attributes (table driven) 

. variables (names, attributes) must be 
converted to machine addresses before 
instructions are selected 

. an IR class will map to a machine storage 
class, e.g., memory, stack, registers, cache 

etc. 

. an IR data type will map to a machine data 
type, e.g., byte, word, long, real etc. 



kinds of Attributed 
Productions 


. Address mode productions. They match 
intermediate form to target machine 
address modes (may or may not generate 
code) 

. Instruction selection productions: represent 
various opcodes that realize an operation. 
These productions are tried in sequence. 
Thus, the last production represents the 
most general case 

. Operand transfer productions: move 
operands to handle conversions, 
destructive operations and non¬ 
orthogonality. (vital to avoid blocking; may 
or may not generate code) 
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Addressing mode 

Addressia - $ Daturmb Datumtc 

displaceib a registers 
ADDRibiCia 

Semantic attributes 

storage class: memory, frame, stack, base register, 

an offset from the base register, 

levels of indirection, 

an index register (if any), 

the datum size and 

tne name of a variable (in case it is global). 
Opcode 

Byterncc - + Byteiliccl Byteinccr 
- Busyir 

EMIT i’incb’iriOiOiCC 
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Transfer Productions 

. Destructive operations: two-address 
instructions 

. Delta-type conversion: mixed mode 
arithmetic 

. instruction-set non-orthogonality:e.g., no 
mem-to-mem arithmetic possible {Zm, 
iAPX-86); no mem-to-mem mul or div (o , 
PDP) 

An example of a transfer production: 


Longincc 


Wordiaicca 

CheckConverUai’word'i’long 

TEMPi’long'rr 

EMIT i’cvtwl’ lairiOiCC 
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;= 40//6-M TADRtBB DATUM f VAL */ 

(.4) MATCH Dofi/btR —* ADRfy CoHVfRTl iy 
£M!T : CVTWl 8B,R 
.♦= LoHCrfA+LoVCrtR OATOMfVAL */ 

(5) 

Hatch ionct'X —> 

PATUMtV isomw v 

HoTBosy^X 
EMIT: JncL R 
;= loHCrfA LoMCr TR 

( 6 ) HATCH IHST-* l = LoH6nLoNC-V 

K WTfC UUy 

EMIT : MOVL R t A 



Register Allocation 

. Invoking action symbol Temp is an 

,on-the-fly call for a temporary 
requirement. 

Gettemp is given storage-class, data-type, 
size and preference (e.g., AX on the 8086, 
DO on the 68k, even register on the PDP) 

TempiAccumulatori’word’iAX 

TempiDataregisteri’long’iDO 

TempiEvenregisten’word’iANY 

. In the absence of Inherited attributes to LHS 
symbols, context of operand use is 
determined by prefix operator in IR and left 
context at constant offset from top of parser 
stack. 

. To guarantee conditions on spills, a 
separate global register allocator is 
required; the intentions are expressed as 
preference attributes in IR 

. Register allocation can also be performed 
post instruction-selection. In this case, 
Gettemp manages pseudo registers instead 
of actual registers. 
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Performance of Retargetable Code Generators 

. Code Generation Speed depends largely on 
pre-analysis. Reported speeds range from 
10 to several thousand instructions/second, 
with speeds of several hundred 
instructions/second most common. 

. The code quality of the portable C compiler 
is comparable to hand-crafted compilers. 

. Other code generators (Graham/Glanville, 
Ganapathi/Fischer) have produced code 
somewhat better than portable C. 

• Retargetable Pascal compilers for Intel and 
Siemens machines have reported superior 
performance to native compilers. 

. Code Generator size depends on tables. 
Parsing-based generators require the most 
space with sizes of 100K or more required 
for UNCOMPACTED tables. 

. Retargeting to a new architecture has been 
estimated to require about one man-month 
(for experienced implementors). 
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Experiences 

. Code generators (or VAX-11/780, Intel 
8086, IBM-370, POP-11 /70, Z-SOOO, 
MC68000 and FOM- 

. size: 100K to 120K mostly tables not 
compacted 

. CGG time taken = 4 mins (real time) 

. CG speed: 100 lines of assembler code per 
second 

. Grammar size (highly optimizing): 

VAX: 242 symbols and 578 productions 

8086: 271 symbols and 558 productions 

370: 200 symbols and 350 productions 

POP: 205 symbols and 346 productions 

Z8k: 90 symbols and 250 productions^ 

68k: 74 symbols and 216 productions* 
FOM: 184 symbols and 267 productions 

XZt: 

. code Quality. -5-10% better size than Unix 
C (compiler + optimizer); worse than 
Bliss-11 



Machine-dependent & Peephole 
Optimizations 

. Constant optimizations and constant folding 

. Strength reduction (e.g., ashl, mull2, mull3 
on VAX) 

. Add productions to model special purpose 
instructions: increment, decrement, 
multiply via shift, subtract, test and 
branch (e.g., aobleq on the VAX) 

. Subsuming code within addressing modes 
e.g., @ + ••• into indexing; + as movab, 
pushab etc.; add or subtract via 
auto-increment/decrement double¬ 
register indexing (e.g., VAX, 68k) 

. Delete redundant code: add or subtract of 0, 
multiply or divide of 1, unnecessary 
condition-code evaluations 

. Instruction buffering and Tracking: Delay 
code generation by buffering: delayed 
moves, remember register contents 
(tracking), hoist loads 



A u tomatic Pe ePh/ot e 
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EjAflPie: suB ^f2 J R3 
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TH/S Foftti oF p.o. ca/o 

SLOW ^Lojs <$F MA HI Pula T*oa/ 
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